3

I am trying to revamp my legacy application to make it scalable and performant. Its current architecture is something like this

Consider a short-lived script that gets invocated 500k+ times a day, each invocation is a unique invocation (identified by a Key) and it writes a couple of structured files to its own unique dated directory. There could be script reruns too (reruns would update the files in the same partition).

Now I have a Web application to show information from the execution of this script (data persisted by the script runs in the file system) in UI.

The backend of this web application is Java-based. It has 7days of in-memory cache (hashmaps) with dedicated threads that wake up every 30 sec and refresh the data in cache by reading fresh information from the file system. Note that in-memory cache with 7days of data takes around 40Gb of RAM space.

The frontend is react based. We refresh data in the browser by querying data from the Java backend every 30 sec by making an API call.

As you could notice there are three main issues with this architecture:

  1. There could be a delay of 30sec(backend refresh) + 30 sec(frontend refresh) in showing fresh data in the UI. This is because we are polling data and pushing data on updates.
  2. Since we have an in-memory cache, it is not possible to scale this application horizontally by replicating server instances. We will end up having multiple caches across different instances of server and each will have its own refresh cycle.
  3. Queries before 7 days are too slow because data is not available in the in-memory cache of the Java server. We have to read it from the file-system on the fly.

How can I improve this architecture?

One possible Architecture:

I was thinking about introducing a Kafka Queue where scripts can publish events along with writing to a filesystem. Java server can subscribe to these Kafka events. On receiving events from the Kafka, the Java server can

  1. Update data to the Redis Cache
  2. Persist it to the database, and
  3. Push updates to UI over WebSockets

Does this sound good or you see any flaw?

Improve this question
7
  • 4
    Why are you storing the data in the filesystem and not in a database? Is there any chance to revisit that decision?
    – Bart van Ingen Schenau
    Mar 30, 2020 at 8:46
  • 2
    One possible improvement is not to run a script half a million times a day, creating a million files a day.
    – Steve
    Mar 30, 2020 at 8:46
  • @BartvanIngenSchenau I am refactoring the web application. Changing the script invocation is not under our control.
    – ThinkGeek
    Mar 30, 2020 at 13:45
  • I have updated the question with one possible Architecture that came to my mind. Let me know if you think there are any flaws.
    – ThinkGeek
    Mar 30, 2020 at 14:31
  • @BartvanIngenSchenau Information cannot be stored in the database as of now.
    – ThinkGeek
    Apr 2, 2020 at 6:47

2 Answers 2

Reset to default
2

A potential solution is to use a database instead of the raw file system. With indexes built on your expected query patterns, you should see more reliable performance than just dealing with raw files.

Since you're refactoring to this slowly, you may need to duplicate the storage mechanisms and grow it over time. That is, write your data to both the existing file system, and also to the DB. Then, ideally, you can slowly move your queries over from one data source to the other. The downside/challenge here is keeping the two data stores in sync with each other.

You can also consider something like Redis for your in memory cache layer, as a mechanism to get horizontal scalability for your caching layer. This might help you cache more than 7 days worth of data using multiple machines.

Another suggestion might be to write through your cache, so the data is immediately cached in your caching layer, and you don't need to wait for your refresh functionality to kick in. Some data fabrics like Apache Ignite support write-through caching.

Improve this answer
3
  • What about publishing state change events from Script to Kafka, Java server listening to these events and persisting to Database and Copying to Redis Cache too.
    – ThinkGeek
    Apr 3, 2020 at 5:27
  • The idea here is, I do not want to increase overhead on the script. Script writing to FS and also Writing to DB can potentially degrade scripts performance
    – ThinkGeek
    Apr 3, 2020 at 5:28
  • Yes, you can use kafka (or some other message queuing technology) to make your writes asynchronous to lower the performance degradation of the script publishing the events. The downside is that adding more technologies to solve your problem increases the complexity of the overall solution, so you need to be sure that it's worth it.
    – Oleksi
    Apr 3, 2020 at 16:01
2
+50

Regarding your architecture approach at the end of the question:

First of all, we are talking about 500.000 invocations, i.e. one invocation every 0.1728 seconds on average.

As far as I understood, Kafka shines in distribution of live streaming data. I think, if you need guaranteed delivery, you'd be better of with a "classic" message broker like ActiveMq. The throughput it definitely there, and you can persist messages on disk, if the listener / java backend is down for any reason. (Note that this also might be true for kafka, which I never uses in real live. - Just check it.)

Using a redis cache seems a sound idea, but you'll have to persist the cache on disk to cover the scenario of the java backend restarting. I honestly cannot see why redis (with disk storage) is OK for you, but a database is not. Both will work.

Pushing to the client over websockets is fine (or use SSE instead), but not every 0.1728 seconds! You'll have to aggregate your messages and - depending on the message size - use larger intervals for client updates. Whether this will be 1 second, 10 seconds, 1 minute, 1 hour depends on the concrete data, its size, and its use.

Improve this answer
8
  • Choices: Script directly persisting to DB and to Redis Cache vs Script publishing to message broker, Java server listening to events and then Java server persisting date to DB and updating Redis. Can you shed a bit of light on this too? Note that as I mentioned before I decided to publish to Message broker from script because I do not want to add any overhead to the script. If I write to DB and Redis from script itself then I will have to deal with retries, synchronous write operations, etc. Do you think message broker choice is right or script can directly write to DB?
    – ThinkGeek
    Apr 3, 2020 at 6:49
  • 1
    If the script publishes to the message broker, further processing will be asynchronous, i.e. the overhead in the script will be minimal, whereas the "new load" of maintaining caches and persistent copies of these caches will be in the message broker and java server. As the script seems to be the "critical" part of the whole application, this is the way I'd go.
    – mtj
    Apr 3, 2020 at 7:05
  • But what if the message broker goes down? If the operations to publish the event to the message broker are async from the script then I will lose all the messages that were sent to the broker in the interval when the broker is down. In this case, I will end up showing a false state from the Java server. How do we generally handle this?
    – ThinkGeek
    Apr 3, 2020 at 14:41
  • I am fine with DB. My point is I cannot get rid of the file system. I can create a parallel DB store as suggested by Oleksi. Maybe I need both Redis (with disk writes) as well as DB. In Redis, I can store say 2 months of data and the rest of the historical data can there be in DB. My second question is why not eliminate the message broker all together and expose HTTP Post endpoint from the Java server. i.e. script on state change does a POST call to the Java server, and Java server writes to the DB, updates Cache and updates, clients, over a WebSocket connection. How does this sound?
    – ThinkGeek
    Apr 4, 2020 at 4:37
  • 1
    You can use ready made, widely used systems as a broker - Kafka, Rabbitmq (more of a message queuing & distribution) etc depending on the requirement. There are established practices to scale them & deploy for redundancy - or throw money at the problem & get prof. support for them. This makes your java server simpler & to the point of solving only your business problem.
    – asr9
    Apr 6, 2020 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.