I am working on converting an existing python based monolith solution to a microservice. The current flow is pretty straight forward:
Accept XLSX as input -> Run some complex algorithms based on input and a default file(XLSX)-> generate XMLs
The same application cater to multiple projects. The default files are project specific and pre-loaded and not very often these files are modified.These files are put under project specific directory manually upon receiving the files from the users via email.
When a request comes for a particular project, current application reads both input as well as default file to run through the algorithm and finally creates the XML output. The default files are read intermittently and not read as a whole, few sheets are read at the time of input validations and other sheets are read at a later point of time.
As a part of my microservice architecture, I have decided to expose endpoints to upload, update and fetch the default files. Since, these default files are rarely changed, I have decided to parse it as soon as it gets uploaded and store it in a REDIS server, to remove the overhead of parsing it during the actual processing, as these files are tend to be quite large in size(~2GB).
So, I am bit perplexed about reading the default file containing lot of sheets at once and store it in REDIS would be a better solution or reading them sheet wise on an "as-needed" basis would be the one.