Scenario: I am designing a system where user uploads a structured data file ~0-5GB in size. I need to let user analyze this file and perform some transformations on the file which essentially means running functions that extract information from this file and/or change structure/data in the file. User can perform these operation in any order as many number of times as possible. For a start I am evaluating if it is possible to give user a real time (excel like) experience meaning actions are happening instantaneously.

Option 1: One simple option is to store this data into a database and convert user actions into queries and execute it against the database. It has several issues (1) some operations won't be directly available in the database (2) I/O cost for db operations (3) for non-db operations, will have to bring data into memory, execute operation, write back. All of these will take away the instant user experience I am targeting.

Option 2: Is there an in memory database, e.g. memsql, worth considering in this case? Idea is that all the data is in memory and operations would be much faster. Downside is that I will need huge memory, if there are even 5-10 customers you can imagine the RAM requirements.

Option 3 and Question: I don't have much experience with dockers but is following path worth exploring: As soon as a user session starts, I spawn a docker container(on some managed IAAS) with large enough RAM and bring user data into container's RAM. All the user operations happen in RAM (through code) and once user logs off, dump the data back in the datastore. Does this make sense? Is docker spawn time small enough for a case like this?

  • In practice it is likely that most of the data sits in the page cache so performance wise all 3 options are likely to be similar. But if you care that much, benchmark Sep 16, 2017 at 12:57
  • @BasileStarynkevitch but page cache would be limited to size of RAM, right? if three customers, of 15GB total data, are online and RAM is e.g. 12 GB, page cache won't work, and I/O operations will be involved, correct? Sep 16, 2017 at 14:23
  • Have you looked into "serverless" architecture? Something like AWS Lambda? It's designed to be an on demand compute resource.
    – RubberDuck
    Sep 16, 2017 at 14:28
  • @RubberDuck I have but not sure how can it solve my problem, may be I am missing something, can you point to some sample/tutorial/explanation on how it can help in this case? Sep 16, 2017 at 14:32
  • 5
    @HarisHasan if I read your question correctly, you need bursts of large amounts of RAM, but don't want to pay for that RAM all the time you're not using it. That's exactly what Lamda (or Azure Functions) can do for you. They essentially spin up a container when the client calls the endpoint, then toss it away once it's done computing. The difference is you don't have to worry about managing the containers yourself.
    – RubberDuck
    Sep 16, 2017 at 14:47

1 Answer 1


What you can use is AWS Lambda. This lets you run code without thinking about servers. You only pay for the compute time you consume. These guys have done a good job by making it easy to simulate Lambda environment using Docker, have a look https://github.com/lambci/docker-lambda. Different runtime languages are supported.

Some people have proved that you can run Docker containers on AWS Lambda → https://hackernoon.com/how-did-i-hack-aws-lambda-to-run-docker-containers-7184dc47c09b.

You mention a file anywhere from 0 to 5 Gb. Lambda supports up to (just under) 3 Gbs. Check the documentation on the projects for file size. I have not personally tried this on my data.

Take a look at the two projects for testing and prod considerations.

  • 1
    Lambda can use a maximum of (just under) 3 GB of RAM. That feels problematic to me for working on a data set which might be 5 GB in size. Could you give some references to people using Lambda with that size of data? Dec 30, 2017 at 16:55
  • edited the post see above.
    – Lena Weber
    Jan 2, 2018 at 14:21
  • So what makes you believe that Lambda is an appropriate solution? Jan 2, 2018 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.