1

I'm dealing with an application that does the following:

  1. we have a microservice running on EKS called by jobs that pull XML data off an external system. The data volume can be fairly large (700MB-1GB at times). We cannot control the volume of data, there's no pagination or way to pull a delta.
  2. the data gets written to S3 in chunks, one file per record; a message is the sent to SQS queue, triggers lambda function that invokes another service
  3. the next service performs data transformation

The problem we run into is that since we may end up with 1500+ files in S3, writing and reading those is not particularly fast. Also, we're dealing with a situation where there could be dozens of jobs running at the same time, though not all of them would be pulling 1GB, but you could have a number of them that do. We've thought of a few ways to try to remedy:

  1. use ElastiCache/Redis; my concern here is that we'll need too much memory and it'll be cost prohibitive. If we keep the memory on the low side, it means we need to be able to free it up fast, but running transformations on 1500 files is not going to necessarily be that fast, so we could end up holding memory for a minute or two, which in this case is quite a long time.
  2. use DynamoDB; concern is the 400KB limit, however since it's text we may be able to get away with it using compression?
  3. use a relational database; there's no relational data here, but given #1 and #2 seem to have issues and we don't need lightning fast reads/writes (just needs to be faster than S3), seems like this might be able to fill the gap. There shouldn't be any concurrency/blocking issues as one service only writes and the other only reads, and there aren't even multiple reads to the same record.

I'm sure there are options I haven't thought of, possibly even services I don't realize I can leverage. Constraints here is I can't redesign the way this whole thing works (though I won't mind if people have suggestions for a future when we could refactor) and I'd prefer to stick to using managed services (so no MongoDB or other unsupported AWS data stores) as we're a small team and want to focus on what we're good at and not managing infrastructure.

7
  • maybe SQS? aws.amazon.com/about-aws/whats-new/2015/10/…
    – Ewan
    Dec 16, 2021 at 17:32
  • @Ewan The problem with SQS is that it would probably require us to read a very large amount of data all at once, increasing the memory requirements of the container. By writing a single record to S3 (or wherever else), we keep memory consumption at a minimum. We do already use SQS to send a message to the next service and give it an ID pointing to the S3 folder where the data resides so the next service knows what to grab. But it can grab and process one file at a time, minimizing the memory footprint.
    – Rocket04
    Dec 16, 2021 at 18:20
  • looking at the 2gb limit it seems like it just stores the message on S3 anyway
    – Ewan
    Dec 16, 2021 at 18:45
  • given that you have the message in memory when you write it to s3, why not send it to the service directly via http at that point?
    – Ewan
    Dec 16, 2021 at 18:46
  • are you using multiple connections to s3?
    – Ewan
    Dec 16, 2021 at 19:04

1 Answer 1

0

Have you considered using EFS? You could try mount the same volume on to all processes (including lambda) that handle the data.

Now, my experience with EFS is not much, since I've used it only once as a case study for a specific problem that needed to get around Lambdas memory limits. It didnt work in my case, since it added a significant latency to the process, as its a File System mounted over a network.

However, it might work for your case. Haven't done a speed test, but I'm pretty sure EFS outperforms S3 in terms of iops. Just make sure you choose the right mode. General tends to be faster then maxIO, but that all depends on your data.

1
  • Thanks for that. Had forgotten about EFS, and it could definitely be a solution. And since we shouldn't be dealing with a lot of frequently accessed data, the cost factor should be pretty favorable compared to other options.
    – Rocket04
    Dec 22, 2021 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.