1

Usecase: End user places an order on the company website, the order needs to be pushed to company admin's screen with small latency.The company has multiple branches(identified by compId and branchId combination) so the order must be pushed to correct admin.

What I have so far: A REST API that takes Order, persists it to MongoDB and writes the message to Messaging gateway. The messaging gateway decouples my email sender service from my web app.

What I am struggling with: How to push orders to browser based admin client where delivery is guaranteed and in real time? I am thinking about using long polling for admin client, as the volume of orders is not big enough to justify WebSockets(Is this correct?). And Apache Kafka Or Redis for guaranteed delivery and real time order streaming on the server side.

Solution I have come up with: When the admin logs-in SseEmitter is created on the server and stored in a Map where the key is compID and BranchId combination. When an order comes in it is added to Kafka topic or Redis. A listener(kafka or redis)listens for these events and tries to get the SSE for this order from the Map if found write the order to the SSE. And remove the order from the topic or Redis DB.

Questions:

  1. What is better suited for this Kafka or Redis? I have no prior experience for both.
  2. What could be possible problems with this approach? long polling, messaging, etc. Is there something I am over looking?
  3. Is my assumption regarding web sockets correct?
  4. Any other insight you may have.
  • 4
    I would think web sockets would be fine. Some clients offer a long polling fallback if web sockets aren't supported. – GrandmasterB Jan 26 '17 at 4:58
  • @GrandmasterB Do you have any library in mind? – Rohit Jan 26 '17 at 21:45
  • The question is more likely: AMQP vs Streaming. It might interest: RabbitMQ vs Kafka. I agreed with @NikosKos, I think AMQP will do the work. – Laiv May 27 '17 at 14:15
  • 1
    For me the elephant in the room is what does 'REAL TIME' mean? I have a feeling we are probably not talking the usual CS definition here (Bounded latency). Designing something to have 100% reliable function where a failure to meet a latency target is a hard failure is I suspect not really the target here. Does "Real Time' really just mean 'quickly on a human timescale' to you, as opposed to provably guaranteed within X microseconds? Adding that guarantee makes the whole problem MUCH harder, particularly in a GCd language not designed for that sort of thing. – Dan Mills Jul 12 '18 at 13:31
2
  1. Kafka and redis are not used for the same purpose. Redis is a key-value storage engine and Kafka is "streaming platform". I have never used Kafka but it seems to act like a message broker optimized for streaming

We have come across a similar demand at the company I work for. We had to mark seats as occupied on a seating plan along with people having their tickets scanned at the entrance of the theater. Here is a simplified version of our solution :

  • you need a message broker (rabbitMQ), a nodeJS daemon, a websocket client.
  • whenever a ticket is scanned, the corresponding data is sent to rabbitMQ thus creating a new message in a queue
  • the nodeJS daemon constantly checks the rabbitMQ queue for new messages.
  • whenever a supervisor connects to the display interface, a websocket is open with the nodeJS daemon. This websocket will be used to notify the client that some new data has to be fetched on the server.
  • whenever a message arrives in the queue, it is read (and destroyed to prevent it from being processed twice) by the nodeJs daemon. The daemon notifies every connected client through the websockets
  • whenever a client receives a websocket notification, it polls the fresh data from the webserver.
  • Thanks for the answer. I was thinking of Redis for low latency inter process communication(yes its bit of hack). I can see the similarties between our UCs, but there is difference. when the supervisor is not logged in the seating plan is updated and he/she can see it later. In my case admin is not logged in the message cannot be processed. What client library you used for WS? – Rohit Jan 26 '17 at 23:24
  • "In my case admin is not logged in the message cannot be processed" <-- cannot or should not ? The client library we use is socketIO. – NikosKos Jan 27 '17 at 7:45
0

I don't think this is challenging as some are making it out to be. Your question is tagged Java, so I am going to work with that.

First, since this is dealing with orders, and not reacting to automotive traffic, I am going to assume a latency of a second or two is acceptable and subsitutable for real time.

I would say, just use Redis, and a Java web app; keep it simple. Websockets are just not as reliable as more mature technologies, and I doubt "push" notifications are really needed. I am supposing you don't have more than a few thousand admins, right?

Redis is simple, highly available, has its own queue implementation, and is blazing fast.

While you certainly could use mongo, I think a regular sql DB would be more appropriate; I very much suspect your data is relational. You can use redis to make things fast, not that a properly designed DB is slow.


When a order is placed, its summary is placed in a redis queue. It is then, also asynchronously in your primary db (I highly recommend you use postgres or oracle for your main db).

Your admin webpage is a simple javascript timer, firing maybe every half second or so, polling a simple json api made in a spring controlle . This json api checks the redis queue and sends minimal json response to the admin script (perhaps just order id and title). If the response is not empty, the script redirects to a more in-depth order page, or what ever you need it to do.

If you need more complex queuing behavior, rabbitmq is good, but stick to just redis with a transnational db backing it if you can.

There is no need to overthink this; don't get swept up in your PMs excitement.

  • Note that the assertion that Redis is “blazing fast” is a highly subjective opinion. For the scale of systems I am involved with the performance of Redis is woefully inadequate, so it just depends what you are building. – bikeman868 Aug 11 '18 at 6:38
  • @bikeman868 of course. Although, I would say for the vast majority of webapps, that statement holds true. Redis is single threaded, so as things start to scale, you have to start changing things. Just curious, what solution are you using? – TheCatWhisperer Aug 13 '18 at 13:05
  • 1
    After battling with Redis etc for way too long, we eventually built an in-house equivalent that is more than 100x faster than Redis for our particular scenario. – bikeman868 Aug 13 '18 at 15:52
  • @bikeman868 now im curious what your usecase is =) Any details you can give? – TheCatWhisperer Aug 13 '18 at 16:56
  • 1
    I used it to store session data for each user. We do not use sticky sessions so the session needs to be accessible and updatable from multiple servers. The session must expire after a configurable interval, and is deleted on logout. I also wanted to be able to serve multiple requests to the same user at the same time without locking the session. We have about 100M registered users and serve about 50K requests per second. – bikeman868 Aug 14 '18 at 2:43
-1

Creating our own architecture may be good for maintenance. I think we can make a service oriented architecture with REST + angularjs and also websocket api to push realtime data to the ui. At the same time there are also some java frameworks like webfirmframework which provides guaranteed realtime ui updates without any trigger from ui. Apache Kafka is a distributed messaging system which can reliably collect messages from multiple sources and reliably send messages to billions (or more than that) of subscribed consumers because of its distributed nature it can handle heavy load. Kafka alone will not be enough for displaying message in the browser page, from server to client (browser page) data pushing we need the support of websocket or any other methods like long polling etc..

Redis is an in-memory data storage which can be used as a database, a nosql db, it supports various data structures like, set, map, list etc..

We can also consider RethinkDB, referred from its website : it continuously pushes updated query results to applications in realtime. So, RethinkDB + webfirmframework combination will be an easy way for realtime web app architecture.

-1

Maybe feathers.js (based on express) can help you. With feathers you can easily implement CRUD/Rest Apis which store their data in your favorite db(mongo is officially supported via mongoose). On Top of that it can automatically send these CRUD event notifications to subscribed clients via socket.io. For that, they have an easy to setup client which does most of the work.

protected by gnat Sep 14 '17 at 18:37

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.