Storing large data shared between microservices running in containers at AWS

Question

I'm dealing with an application that does the following:

1. we have a microservice running on EKS called by jobs that pull XML data off an external system. The data volume can be fairly large (700MB-1GB at times). We cannot control the volume of data, there's no pagination or way to pull a delta.
2. the data gets written to S3 in chunks, one file per record; a message is the sent to SQS queue, triggers lambda function that invokes another service
3. the next service performs data transformation

The problem we run into is that since we may end up with 1500+ files in S3, writing and reading those is not particularly fast. Also, we're dealing with a situation where there could be dozens of jobs running at the same time, though not all of them would be pulling 1GB, but you could have a number of them that do. We've thought of a few ways to try to remedy:

1. use ElastiCache/Redis; my concern here is that we'll need too much memory and it'll be cost prohibitive. If we keep the memory on the low side, it means we need to be able to free it up fast, but running transformations on 1500 files is not going to necessarily be that fast, so we could end up holding memory for a minute or two, which in this case is quite a long time.
2. use DynamoDB; concern is the 400KB limit, however since it's text we may be able to get away with it using compression?
3. use a relational database; there's no relational data here, but given #1 and #2 seem to have issues and we don't need lightning fast reads/writes (just needs to be faster than S3), seems like this might be able to fill the gap. There shouldn't be any concurrency/blocking issues as one service only writes and the other only reads, and there aren't even multiple reads to the same record.

I'm sure there are options I haven't thought of, possibly even services I don't realize I can leverage. Constraints here is I can't redesign the way this whole thing works (though I won't mind if people have suggestions for a future when we could refactor) and I'd prefer to stick to using managed services (so no MongoDB or other unsupported AWS data stores) as we're a small team and want to focus on what we're good at and not managing infrastructure.

Answer 1

Have you considered using EFS? You could try mount the same volume on to all processes (including lambda) that handle the data.

Now, my experience with EFS is not much, since I've used it only once as a case study for a specific problem that needed to get around Lambdas memory limits. It didnt work in my case, since it added a significant latency to the process, as its a File System mounted over a network.

However, it might work for your case. Haven't done a speed test, but I'm pretty sure EFS outperforms S3 in terms of iops. Just make sure you choose the right mode. General tends to be faster then maxIO, but that all depends on your data.