Skip to main content
28 votes

Message Queue. Database vs Dedicated MQ

Everything I have read on the internet suggests that using a database for a Message Queue isn't a scalable solution. The reality frequently ignored by the "don't use X because it doesn't scale" (link ...
Blrfl's user avatar
  • 20.4k
11 votes

Message Queue. Database vs Dedicated MQ

Message Queues really come into their own when you have many of them and route messages between them, fanout to more than one consumer etc. If you just have a single 'job queue' of stuff you want to ...
Ewan's user avatar
  • 77.4k
11 votes

How to optimize average rating calculation in a review system?

But assuming it's a large-scale application, updating the average each time would not be a good idea, because it would be a significant DB load to fetch all reviews and make average of them each time (...
Doc Brown's user avatar
  • 210k
10 votes

Message Queue. Database vs Dedicated MQ

As other have mentioned scale is probably not important here. The problem with using two different storage mechanisms is transactional integritety. If going with a dedicated message queue you need to ...
Esben Skov Pedersen's user avatar
10 votes

How can message queues improve scalability?

It's not that queues are more scalable, its the fact that two services communicating through queue means the communication is asynchronous. Asynchronous communication is far more scalable than ...
Matan Shabtay's user avatar
8 votes
Accepted

How to implement a message queue over Redis?

If you want to use Redis for a message queue in Node.js and you don't mind using a module for that then you may try RSMQ - the Redis Simple Message Queue for Node. It was not available at the time ...
rsp's user avatar
  • 7,858
8 votes

Implementing a caching microservice by avoiding potential bottlenecks

My suggestion is that instead of worrying about whether to use an external or internal cache, your first concern should be that your booking-service does not care whether or not your are using an ...
TheCatWhisperer's user avatar
8 votes

Traditional Message Brokers and Streaming Data

Kafka/Kinesis is modelled as a stream. A stream has different properties than messages. Streams have context to them. They have order. You can apply window functions on streams. Although each item in ...
Samuel's user avatar
  • 9,197
8 votes
Accepted

Traditional Message Brokers and Streaming Data

Kafka deals in ordered logs of atomic messages. You can view it sort of like the pub/sub mode of message brokers, but with strict ordering and the ability to replay or seek around the stream of ...
closeparen's user avatar
7 votes
Accepted

AMQP routing-keys naming anti-patterns when using topics

It seems like a mistake to me. The point of publishing a message is that you are decoupled from the component or components receiving the message. The business logic about whether the message is then ...
Ewan's user avatar
  • 77.4k
7 votes

Message Bus v Mediator pattern v In Memory Bus

It's not that simple, and of course "it depends": I can't remember where, but I think I remember reading Roger Johansson write something along the lines of "Don't queue messages using ...
snowcode's user avatar
  • 171
7 votes

Dealing with data arriving at a different times

Don't view the result as the outcome of a request, but as the followup of an event. The question doesn't specify much so I can only speak in abstract terms. I believe the following to be the best ...
Lacey's user avatar
  • 161
7 votes

Event bus vs PubSub

For the "publisher/subscriber" pattern, subscriber components have to subscribe (or unsubscribe) to an event publisher. This means the life time of the event publishers is as least as long ...
Doc Brown's user avatar
  • 210k
6 votes

Message Queue. Database vs Dedicated MQ

For practicality, main reasons for me to use a message queue are: I didn't have to reinvent the wheel. Designing even the simplest message queue using database requires at least a couple of hours of ...
imel96's user avatar
  • 3,518
6 votes
Accepted

Message Bus v Mediator pattern v In Memory Bus

A mediator pattern creates a flexible decoupled interface between two microservices. Messages are sent to and from each microservice without one necessarily having to know all the explicit workings ...
Neil's user avatar
  • 22.8k
6 votes

What can I do to get a message processor to slow down the rate of writes that it is making to a database?

You're on the right track with having the system adapt to current conditions. If a database is already busy, asking it how busy it is just adds one more thing for it to do. Instead, flip the ...
Greg Burghardt's user avatar
5 votes

Solve Synchronization

As per the wikipedia page you linked to regarding message queues: Message queues provide an asynchronous communications protocol, meaning that the sender and receiver of the message do not need ...
Ben Cottrell's user avatar
5 votes
Accepted

Should we be using pub/sub in our messaging stack?

I have written a similar system about a month ago, and just like you I've discovered that this is best solved by a reliable queue technology with a redrive. While some might argue using SQL as your ...
zaitsman's user avatar
  • 384
5 votes
Accepted

How to deal with failing messages in DDD?

One way to deal with this is to use pull-based listeners instead of push-based. Each listener keeps track of its last read message and can request "messages since X". You could use either polling or a ...
Kasey Speakman's user avatar
5 votes

Communicating bulk data among microservices

I work on a system that has similar requirements - partners drop files that contain anywhere from 10 - 1,000,000 records to be processed. Some partners drop the files once a day, some once an hour, ...
sceaj's user avatar
  • 304
5 votes

How to handle data inconsistency in microservice architecture?

I believe there's three typical ways you could handle this: Guarantee delivery of your messages Run a reconciliation process Switch to a pulled events approach A little detail on each approach... ...
Graham Lea's user avatar
5 votes
Accepted

Distributed message queue, propagating queue leader/follower information

The problem with the last option is that I would still need somewhere to store queue metadata information like creation date. Other than you may store other fields (like timestamp) in Metadata ...
lennon310's user avatar
  • 3,222
5 votes

Long polling and message brokers

HTTP long polling is just a buzzword for opening an HTTP connection to a web server, and keeping it open in order to repeatedly receive chunks of data. This was a workaround for web browsers that did ...
Greg Burghardt's user avatar
4 votes

Message persistence in Message Queues

Consider a client that wishes to push a message to a queue. The queue manager would typically send a message acknowledgement to the client to signify that the message was successfully delivered to ...
maple_shaft's user avatar
  • 26.5k
4 votes
Accepted

Best practice for delaying jobs

Your two main concerns here in high level design are essentially, the messaging from one or more software components to other software components, and flexible scheduling of jobs based on the age of a ...
maple_shaft's user avatar
  • 26.5k
4 votes

How to implement "viewed: N times" functionality for an article?

I would suggest updating the counter on each view. As @amon suggested - that hard part is determining WHEN to update the counter. An old adage - no premature optimization before its time. ...
ripvlan's user avatar
  • 141
4 votes
Accepted

How do I set up short-lived queues?

ActiveMQ does support this and it's a general feature of JMS. Here's a link pointing to a way of doing this. I can't find the specific documentation at the moment but my recollection is that you ...
JimmyJames's user avatar
  • 27.8k
4 votes

When consuming an api, what is a good way to deal with their request limits?

If the service works by rejecting only requests that are above the limit, and not just killing the whole service for you, then you could implement similar algorithm to how TCP works. Simply said, ...
Euphoric's user avatar
  • 37.5k
4 votes

Synchronization of data across microservices

I would take a look at one of Microsoft's newer projects code named "Ambrosia" (link will take you to their Github page where the project is being developed open source) which focuses on providing a ...
Chaplin Marchais's user avatar
4 votes
Accepted

Architecture for message processing with scheduling, at scale

Kafka does not support scheduled delivery, AWS SQS may be an alternative. Otherwise you have to set up the schedule on consumer side, like you mentioned APScheduler or something similar. Even if you ...
lennon310's user avatar
  • 3,222

Only top scored, non community-wiki answers of a minimum length are eligible