3

I am designing the architecture of an application which has your typical backend and frontend components. The frontend application must be semi real-time. For this requirement, the backend is pushing messages on a queue and the frontend is consuming that queue. Most likely, I will be using RabbitMQ for this. Let me explain.

When the user is logged in into the frontend application, a session is created in the backend. As long as this session is valid, the frontend application is allowed to operate. Typically, after the log-in, the frontend initially loads all data from a REST API (with his session as authentication credentials) and all subsequent updates will be received via the message queue.

In my current design, each frontend session has its own queue. So whenever something changes in the backend, a message is sent to all relevant session queues. As said, the frontend is consuming this queue.

The problem is when the frontend is disconnected from the queue. For example, when the user hibernates his laptop at the end of his working day. From then on, messages starts piling up. The real problem is here that the queue becomes populated with duplicated messages. The longer the client is disconnected from the queue, the bigger this problem becomes.

Small side-note, with duplicate messages I actually mean semantically identical messages. Take a stock exchange application for example. Two price of stock xyz has been changed messages are semantically identical. They come from a different event/transaction, but define the same change. So, the previous message has become obsolete by the new one. See comments for more detail.

I am looking for an architecture to solve this. Let me go over my thoughts:

  • Instead of having a session queue, I could have created a session exchange. After the log-in, a dedicated, auto-delete, private queue could be made which is bound to that exchange. After the disconnect/timeout, this queue will be deleted by the broker. Then, when the frontend application comes back, a new queue will be made. In other words, the frontend application returns with an empty queue.

    • But, this forces the client to refresh all data after a reconnect. Everything could have been changed.
    • I would like to keep the client as dumb/simple as possible.
    • Therefore, I believe this is not an optimal solution.
  • So, it would make sense to solve this in the backend. In other words, I need a way to prevent duplicate messages on the session queues. If a certain message is already in the queue, I will not queue the same message again.

    • What are my options here? How do I find out that a certain message is already in the queue? I perfectly understand that this is not directly supported by AMQP. So, to implement this on my own, I believe this means means I should be able to keep track on message delivery and ideally also on message acknowledgements. Is this possible?
    • Another route could be to make us of the TTL, dead-letter, ... functionalities of the (Rabit)MQ broker. However, I could not come up with something that really works.

My question boils down to this: how do I get the client up to date after a disconnect?

Any ideas or insights are very welcome.

  • Can you elaborate how duplicate messages can accumulate in the queue? If the update messages contain all the changes/new values involved in the update, I don't see how you can get duplicate messages. – Bart van Ingen Schenau Jun 28 '17 at 8:04
  • @BartvanIngenSchenau Assume a stock exchange application. First message: Price of stock XYZ has changed. Two minutes later, again: Price of stock XYZ has changed. When the frontend connects again, he has two refresh twice the price of the XYZ stock. – TheMQJuggler Jun 28 '17 at 8:08
  • So, I meant semantically identical messages. I will add that to the question. – TheMQJuggler Jun 28 '17 at 8:10
  • 1
    They are not really identical messages. But the point that I get is that the older messages are no longer relevant, because the front-end is only interested in the most recent values. That might not be the case if, for example, you wanted to show a graph of how the stock price changed over time. – Bart van Ingen Schenau Jun 28 '17 at 8:53
  • @BartvanIngenSchenau Exactly, you are right. I meant to say that the old ones are indeed not relevant anymore. – TheMQJuggler Jun 28 '17 at 8:55
1

I am not sure to see the values of the queues here, a direct bidrectionnal communication would be better, and would solve your current problem, when the user come back from hibernating his computers, he has to refresh fully his page to trigger back the bidrectionnal communication.

If you use Web application, look for STOMP/Web Socket, if you don't, you can still use STOMP and Web Socket, but a simple TCP Connection could do the work too.

Note that they're libraries to provide very reliable TCP Connection with auto-reconnect and so on to handle network troubles don't go the way to code that again, such things are really a pain to develop and tests properly.

Others way :

  • Instead of delivering the messages one by one to the client, deliver everything when they're more than one, translate them to an Set (collection of unique objects) of actions to do and perform all the Actions.
  • Why the session of the user just don't expires after like 30mn of inactivity ? When it expires clean the queue.
  • RabbitMQ does not implements JMS Spec, so it may not have an equivalent of JMS Topic with durable subscriptions, which seems to fit better than your system.
  • Actually, in my architecture, I am proposing STOMP over WS/SockJS, bound to the RabbitMQ broker. It works indeed as a direct communication socket. But that is not really my question, though. My question is: does it make sens to force the client to fully refresh his page when he comes back? I am not really sure. Therefore, I am looking for a way to notify the client with all relevant updates when he comes back. This means I should avoid duplicate messages. – TheMQJuggler Jun 28 '17 at 8:31
  • Regarding your other ways: (1) I want the application to be (semi) real-time. I want to update the client as soon as possible. (2) I want sessions to remain active overnight, so the user does not have to log-in again when he comes back next morning. (3) I will most likely be using RabbitMQ. I am not familiar with the concept of a durable subscription, I will look into that. – TheMQJuggler Jun 28 '17 at 8:33
  • @TheMQJuggler after verification I mixed RabbitMQ and ActiveMQ, seems like RabbitMQ do not implements JMS spec. So it may not have an equivalent to Topic and durable subscription. Letting a session actives for too long is usually a security risks, this is why you get logged off after some times of inactivity. – Walfrat Jun 28 '17 at 8:46
  • Thanks, I understand. But let's not focus too much on the session lifetime. Sessions will be cleared, of course, but let's assume the client becomes disconnected during the lifetime of his session. When he comes back, he will re-use the session. From there one, my question starts: how do I get the client up to date? – TheMQJuggler Jun 28 '17 at 8:50
  • I already covered that : Instead of delivering the messages one by one to the client, deliver everything when they're more than one[...] of course in that solution you only ignore duplicate on clmient side, you still generate duplicate message. An other way I can think about would be to check if you can implements your own type of Queue for RabbitMQ and make him use it. Then you would be able to not store duplicates. Note : from rabbitmq team mail list : lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-June/… – Walfrat Jun 28 '17 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.