Scenario: I am designing a system where user uploads a structured data file ~0-5GB in size. I need to let user analyze this file and perform some transformations on the file which essentially means running functions that extract information from this file and/or change structure/data in the file. User can perform these operation in any order as many number of times as possible. For a start I am evaluating if it is possible to give user a real time (excel like) experience meaning actions are happening instantaneously.
Option 1: One simple option is to store this data into a database and convert user actions into queries and execute it against the database. It has several issues (1) some operations won't be directly available in the database (2) I/O cost for db operations (3) for non-db operations, will have to bring data into memory, execute operation, write back. All of these will take away the instant user experience I am targeting.
Option 2: Is there an in memory database, e.g. memsql, worth considering in this case? Idea is that all the data is in memory and operations would be much faster. Downside is that I will need huge memory, if there are even 5-10 customers you can imagine the RAM requirements.
Option 3 and Question: I don't have much experience with dockers but is following path worth exploring: As soon as a user session starts, I spawn a docker container(on some managed IAAS) with large enough RAM and bring user data into container's RAM. All the user operations happen in RAM (through code) and once user logs off, dump the data back in the datastore. Does this make sense? Is docker spawn time small enough for a case like this?