0

I am working on converting an existing python based monolith solution to a microservice. The current flow is pretty straight forward:

Accept XLSX as input -> Run some complex algorithms based on input and a default file(XLSX)-> generate XMLs

The same application cater to multiple projects. The default files are project specific and pre-loaded and not very often these files are modified.These files are put under project specific directory manually upon receiving the files from the users via email.

When a request comes for a particular project, current application reads both input as well as default file to run through the algorithm and finally creates the XML output. The default files are read intermittently and not read as a whole, few sheets are read at the time of input validations and other sheets are read at a later point of time.

As a part of my microservice architecture, I have decided to expose endpoints to upload, update and fetch the default files. Since, these default files are rarely changed, I have decided to parse it as soon as it gets uploaded and store it in a REDIS server, to remove the overhead of parsing it during the actual processing, as these files are tend to be quite large in size(~2GB).

So, I am bit perplexed about reading the default file containing lot of sheets at once and store it in REDIS would be a better solution or reading them sheet wise on an "as-needed" basis would be the one.

  • 1
    There is no sensible way to tell you what is "better", as long as you do not define what you mean by "better". And if you mean "faster", or "requiring less resources", the only one who can tell is the person who implements both solutions and measures the differences in performance and resources. – Doc Brown Jan 14 at 20:53
  • What is your performance target? Ie. How quickly should an incoming request be handled? Is there much processing that needs to be done on the default XLSX file to handle a request, or is it just reading it and pulling out the values? Do you need to have Redis running anyways, or would it mean adding it to your stack just for this? – Alex D Jan 14 at 20:53
  • Redis will be running anyway since we are using rabbitmq + celery for the asynchronous communication which inturn using redis to store information. So, i was thinking to use the same redis nstance to store information for all the other services be required. – Pro Jan 15 at 2:20
2

It's not possible to know for sure whether caching your data in Redis would be worthwhile without more information, but:

Don't jump to the conclusion that you need to cache the data in Redis. Every dependency you add to a project has a cost. Once you add Redis, you have added one more potential source of problems. When debugging your program, you will now have to deal with multiple interacting processes rather than just one process. When you move to a new server, there will be more things you have to set up. And so on...

You may or may not find that it actually makes things faster. If you are trying to avoid reading 2GB from disk on every request, consider that your OS does have a page cache, so if your system has extra RAM, it may turn out that the XLSX files are being cached in memory anyways.

If your concern is that parsing XLSX files is slow (I don't know if that is true), you can always convert them to a format which is easier to parse.

Even if you determine that caching is necessary, there is more than one way to do it. For example: will you only be running a single instance of the application which persists between requests? If so, there would be little reason to use Redis for caching (just keep the data in local process memory, using a hash or something). A database is another option, which would look especially attractive if you are already running one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.