Web application Architecture for real time app

Question

Usecase: End user places an order on the company website, the order needs to be pushed to company admin's screen with small latency.The company has multiple branches(identified by compId and branchId combination) so the order must be pushed to correct admin.

What I have so far: A REST API that takes Order, persists it to MongoDB and writes the message to Messaging gateway. The messaging gateway decouples my email sender service from my web app.

What I am struggling with: How to push orders to browser based admin client where delivery is guaranteed and in real time? I am thinking about using long polling for admin client, as the volume of orders is not big enough to justify WebSockets(Is this correct?). And Apache Kafka Or Redis for guaranteed delivery and real time order streaming on the server side.

Solution I have come up with: When the admin logs-in SseEmitter is created on the server and stored in a Map where the key is compID and BranchId combination. When an order comes in it is added to Kafka topic or Redis. A listener(kafka or redis)listens for these events and tries to get the SSE for this order from the Map if found write the order to the SSE. And remove the order from the topic or Redis DB.

Questions:

1. What is better suited for this Kafka or Redis? I have no prior experience for both.
2. What could be possible problems with this approach? long polling, messaging, etc. Is there something I am over looking?
3. Is my assumption regarding web sockets correct?
4. Any other insight you may have.

Answer 1

1. Kafka and redis are not used for the same purpose. Redis is a key-value storage engine and Kafka is "streaming platform". I have never used Kafka but it seems to act like a message broker optimized for streaming

We have come across a similar demand at the company I work for. We had to mark seats as occupied on a seating plan along with people having their tickets scanned at the entrance of the theater. Here is a simplified version of our solution :

• you need a message broker (rabbitMQ), a nodeJS daemon, a websocket client.
• whenever a ticket is scanned, the corresponding data is sent to rabbitMQ thus creating a new message in a queue
• the nodeJS daemon constantly checks the rabbitMQ queue for new messages.
• whenever a supervisor connects to the display interface, a websocket is open with the nodeJS daemon. This websocket will be used to notify the client that some new data has to be fetched on the server.
• whenever a message arrives in the queue, it is read (and destroyed to prevent it from being processed twice) by the nodeJs daemon. The daemon notifies every connected client through the websockets
• whenever a client receives a websocket notification, it polls the fresh data from the webserver.

Answer 2

I don't think this is challenging as some are making it out to be. Your question is tagged Java, so I am going to work with that.

First, since this is dealing with orders, and not reacting to automotive traffic, I am going to assume a latency of a second or two is acceptable and subsitutable for real time.

I would say, just use Redis, and a Java web app; keep it simple. Websockets are just not as reliable as more mature technologies, and I doubt "push" notifications are really needed. I am supposing you don't have more than a few thousand admins, right?

Redis is simple, highly available, has its own queue implementation, and is blazing fast.

While you certainly could use mongo, I think a regular sql DB would be more appropriate; I very much suspect your data is relational. You can use redis to make things fast, not that a properly designed DB is slow.

---

When a order is placed, its summary is placed in a redis queue. It is then, also asynchronously in your primary db (I highly recommend you use postgres or oracle for your main db).

Your admin webpage is a simple javascript timer, firing maybe every half second or so, polling a simple json api made in a spring controlle . This json api checks the redis queue and sends minimal json response to the admin script (perhaps just order id and title). If the response is not empty, the script redirects to a more in-depth order page, or what ever you need it to do.

If you need more complex queuing behavior, rabbitmq is good, but stick to just redis with a transnational db backing it if you can.

There is no need to overthink this; don't get swept up in your PMs excitement.

Answer 3

Creating our own architecture may be good for maintenance. I think we can make a service oriented architecture with REST + angularjs and also websocket api to push realtime data to the ui. At the same time there are also some java frameworks like webfirmframework which provides guaranteed realtime ui updates without any trigger from ui. Apache Kafka is a distributed messaging system which can reliably collect messages from multiple sources and reliably send messages to billions (or more than that) of subscribed consumers because of its distributed nature it can handle heavy load. Kafka alone will not be enough for displaying message in the browser page, from server to client (browser page) data pushing we need the support of websocket or any other methods like long polling etc..

Redis is an in-memory data storage which can be used as a database, a nosql db, it supports various data structures like, set, map, list etc..

We can also consider RethinkDB, referred from its website : it continuously pushes updated query results to applications in realtime. So, RethinkDB + webfirmframework combination will be an easy way for realtime web app architecture.

Answer 4

Maybe feathers.js (based on express) can help you. With feathers you can easily implement CRUD/Rest Apis which store their data in your favorite db(mongo is officially supported via mongoose). On Top of that it can automatically send these CRUD event notifications to subscribed clients via socket.io. For that, they have an easy to setup client which does most of the work.