28
votes
Message Queue. Database vs Dedicated MQ
Everything I have read on the internet suggests that using a database for a Message Queue isn't a scalable solution.
The reality frequently ignored by the "don't use X because it doesn't scale" (link ...
- 20.4k
11
votes
Message Queue. Database vs Dedicated MQ
Message Queues really come into their own when you have many of them and route messages between them, fanout to more than one consumer etc.
If you just have a single 'job queue' of stuff you want to ...
- 79.8k
11
votes
How to optimize average rating calculation in a review system?
But assuming it's a large-scale application, updating the average each time would not be a good idea, because it would be a significant DB load to fetch all reviews and make average of them each time (...
- 214k
10
votes
Message Queue. Database vs Dedicated MQ
As other have mentioned scale is probably not important here. The problem with using two different storage mechanisms is transactional integritety.
If going with a dedicated message queue you need to ...
- 5,158
10
votes
How can message queues improve scalability?
It's not that queues are more scalable, its the fact that two services communicating through queue means the communication is asynchronous.
Asynchronous communication is far more scalable than ...
- 226
8
votes
Accepted
How to implement a message queue over Redis?
If you want to use Redis for a message queue in Node.js and you don't mind using a module for that then you may try RSMQ - the Redis Simple Message Queue for Node. It was not available at the time ...
- 7,868
8
votes
Accepted
Traditional Message Brokers and Streaming Data
Kafka deals in ordered logs of atomic messages. You can view it sort of like the pub/sub mode of message brokers, but with strict ordering and the ability to replay or seek around the stream of ...
- 646
8
votes
Traditional Message Brokers and Streaming Data
Kafka/Kinesis is modelled as a stream. A stream has different properties than messages.
Streams have context to them. They have order. You can apply window functions on streams. Although each item in ...
- 9,217
8
votes
Implementing a caching microservice by avoiding potential bottlenecks
My suggestion is that instead of worrying about whether to use an external or internal cache, your first concern should be that your booking-service does not care whether or not your are using an ...
- 5,281
7
votes
Accepted
AMQP routing-keys naming anti-patterns when using topics
It seems like a mistake to me. The point of publishing a message is that you are decoupled from the component or components receiving the message.
The business logic about whether the message is then ...
- 79.8k
7
votes
Message Bus v Mediator pattern v In Memory Bus
It's not that simple, and of course "it depends":
I can't remember where, but I think I remember reading Roger Johansson write something along the lines of "Don't queue messages using ...
- 171
7
votes
Dealing with data arriving at a different times
Don't view the result as the outcome of a request, but as the followup of an event. The question doesn't specify much so I can only speak in abstract terms. I believe the following to be the best ...
- 161
7
votes
Event bus vs PubSub
For the "publisher/subscriber" pattern, subscriber components have to subscribe (or unsubscribe) to an event publisher. This means the life time of the event publishers is as least as long ...
- 214k
7
votes
Accepted
Handling rate limits / delays in consumers without affecting performance of other operations
I assume there is no penalty for a request that is rejected other than the failure. For example, you’d hope that rejected requests don’t count towards rate limiting.
Create one queue for each key. ...
- 47.5k
6
votes
Message Queue. Database vs Dedicated MQ
For practicality, main reasons for me to use a message queue are:
I didn't have to reinvent the wheel. Designing even the simplest message queue using database requires at least a couple of hours of ...
- 3,578
6
votes
Accepted
Message Bus v Mediator pattern v In Memory Bus
A mediator pattern creates a flexible decoupled interface between two microservices. Messages are sent to and from each microservice without one necessarily having to know all the explicit workings ...
- 22.8k
6
votes
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
You're on the right track with having the system adapt to current conditions. If a database is already busy, asking it how busy it is just adds one more thing for it to do. Instead, flip the ...
- 41.9k
5
votes
Solve Synchronization
As per the wikipedia page you linked to regarding message queues:
Message queues provide an asynchronous communications protocol,
meaning that the sender and receiver of the message do not need ...
- 12.1k
5
votes
Accepted
Should we be using pub/sub in our messaging stack?
I have written a similar system about a month ago, and just like you I've discovered that this is best solved by a reliable queue technology with a redrive.
While some might argue using SQL as your ...
- 384
5
votes
Accepted
How to deal with failing messages in DDD?
One way to deal with this is to use pull-based listeners instead of push-based. Each listener keeps track of its last read message and can request "messages since X". You could use either polling or a ...
- 4,401
5
votes
Communicating bulk data among microservices
I work on a system that has similar requirements - partners drop files that contain anywhere from 10 - 1,000,000 records to be processed. Some partners drop the files once a day, some once an hour, ...
- 304
5
votes
How to handle data inconsistency in microservice architecture?
I believe there's three typical ways you could handle this:
Guarantee delivery of your messages
Run a reconciliation process
Switch to a pulled events approach
A little detail on each approach...
...
- 213
5
votes
Accepted
Distributed message queue, propagating queue leader/follower information
The problem with the last option is that I would still need somewhere
to store queue metadata information like creation date.
Other than you may store other fields (like timestamp) in Metadata ...
- 3,242
5
votes
Long polling and message brokers
HTTP long polling is just a buzzword for opening an HTTP connection to a web server, and keeping it open in order to repeatedly receive chunks of data. This was a workaround for web browsers that did ...
- 41.9k
4
votes
How to implement "viewed: N times" functionality for an article?
I would suggest updating the counter on each view. As @amon suggested - that hard part is determining WHEN to update the counter.
An old adage - no premature optimization before its time. ...
- 141
4
votes
Accepted
How do I set up short-lived queues?
ActiveMQ does support this and it's a general feature of JMS. Here's a link pointing to a way of doing this.
I can't find the specific documentation at the moment but my recollection is that you ...
- 28.9k
4
votes
When consuming an api, what is a good way to deal with their request limits?
If the service works by rejecting only requests that are above the limit, and not just killing the whole service for you, then you could implement similar algorithm to how TCP works.
Simply said, ...
- 37.8k
4
votes
Synchronization of data across microservices
I would take a look at one of Microsoft's newer projects code named "Ambrosia" (link will take you to their Github page where the project is being developed open source) which focuses on providing a ...
- 150
4
votes
Accepted
Architecture for message processing with scheduling, at scale
Kafka does not support scheduled delivery, AWS SQS may be an alternative.
Otherwise you have to set up the schedule on consumer side, like you mentioned APScheduler or something similar. Even if you ...
- 3,242
4
votes
Accepted
Acceptable to use synchronous call to another microservice for time-sensitve state change?
Synchronous is not instantaneous. The network operation still takes just as long, possibly longer because synchronous operations occupy threads longer, which decreases server resources. It's still ...
- 148k
Only top scored, non community-wiki answers of a minimum length are eligible