I am trying to revamp my legacy application to make it scalable and performant. Its current architecture is something like this
Consider a short-lived script that gets invocated 500k+ times a day, each invocation is a unique invocation (identified by a Key) and it writes a couple of structured files to its own unique dated directory. There could be script reruns too (reruns would update the files in the same partition).
Now I have a Web application to show information from the execution of this script (data persisted by the script runs in the file system) in UI.
The backend of this web application is Java-based. It has 7days of in-memory cache (hashmaps) with dedicated threads that wake up every 30 sec and refresh the data in cache by reading fresh information from the file system. Note that in-memory cache with 7days of data takes around 40Gb of RAM space.
The frontend is react based. We refresh data in the browser by querying data from the Java backend every 30 sec by making an API call.
As you could notice there are three main issues with this architecture:
- There could be a delay of 30sec(backend refresh) + 30 sec(frontend refresh) in showing fresh data in the UI. This is because we are polling data and pushing data on updates.
- Since we have an in-memory cache, it is not possible to scale this application horizontally by replicating server instances. We will end up having multiple caches across different instances of server and each will have its own refresh cycle.
- Queries before 7 days are too slow because data is not available in the in-memory cache of the Java server. We have to read it from the file-system on the fly.
How can I improve this architecture?
One possible Architecture:
I was thinking about introducing a Kafka Queue where scripts can publish events along with writing to a filesystem. Java server can subscribe to these Kafka events. On receiving events from the Kafka, the Java server can
- Update data to the Redis Cache
- Persist it to the database, and
- Push updates to UI over WebSockets
Does this sound good or you see any flaw?