28
votes
Message Queue. Database vs Dedicated MQ
Everything I have read on the internet suggests that using a database for a Message Queue isn't a scalable solution.
The reality frequently ignored by the "don't use X because it doesn't scale" (link ...
- 20.3k
18
votes
Does Akka obsolesce JMS/AMQP message brokers?
Actor Model
The actor model is computer science strategy for building applications that handle lots of concurrent computation and stateful processing. It's not the only strategy but it's a very well ...
- 280
18
votes
Accepted
Using flat files vs database/API as a transport between a frontend and backend
Switching to a solution involving databases or the queuing systems mentioned by Ewan would
create dependency on a new, complex system in both backend and frontend
introduce unnecessary complexity and ...
- 306
11
votes
Using flat files vs database/API as a transport between a frontend and backend
I don't think either solution is inheritly a bad practice, so answering which is the best practice may be difficult.
I don't believe the YAGNI principal applies here if you're dealing with scale. "...
- 210
11
votes
Message Queue. Database vs Dedicated MQ
Message Queues really come into their own when you have many of them and route messages between them, fanout to more than one consumer etc.
If you just have a single 'job queue' of stuff you want to ...
- 72.4k
11
votes
How to optimize average rating calculation in a review system?
But assuming it's a large-scale application, updating the average each time would not be a good idea, because it would be a significant DB load to fetch all reviews and make average of them each time (...
- 203k
10
votes
Message Queue. Database vs Dedicated MQ
As other have mentioned scale is probably not important here. The problem with using two different storage mechanisms is transactional integritety.
If going with a dedicated message queue you need to ...
- 5,108
10
votes
How can message queues improve scalability?
It's not that queues are more scalable, its the fact that two services communicating through queue means the communication is asynchronous.
Asynchronous communication is far more scalable than ...
- 126
8
votes
Accepted
How to implement a message queue over Redis?
If you want to use Redis for a message queue in Node.js and you don't mind using a module for that then you may try RSMQ - the Redis Simple Message Queue for Node. It was not available at the time ...
- 7,858
8
votes
Implementing a caching microservice by avoiding potential bottlenecks
My suggestion is that instead of worrying about whether to use an external or internal cache, your first concern should be that your booking-service does not care whether or not your are using an ...
- 5,251
8
votes
Traditional Message Brokers and Streaming Data
Kafka/Kinesis is modelled as a stream. A stream has different properties than messages.
Streams have context to them. They have order. You can apply window functions on streams. Although each item in ...
- 9,167
8
votes
Accepted
Traditional Message Brokers and Streaming Data
Kafka deals in ordered logs of atomic messages. You can view it sort of like the pub/sub mode of message brokers, but with strict ordering and the ability to replay or seek around the stream of ...
- 646
7
votes
Accepted
AMQP routing-keys naming anti-patterns when using topics
It seems like a mistake to me. The point of publishing a message is that you are decoupled from the component or components receiving the message.
The business logic about whether the message is then ...
- 72.4k
7
votes
Message Bus v Mediator pattern v In Memory Bus
It's not that simple, and of course "it depends":
I can't remember where, but I think I remember reading Roger Johansson write something along the lines of "Don't queue messages using ...
- 171
7
votes
Dealing with data arriving at a different times
Don't view the result as the outcome of a request, but as the followup of an event. The question doesn't specify much so I can only speak in abstract terms. I believe the following to be the best ...
- 161
6
votes
Message Queue. Database vs Dedicated MQ
For practicality, main reasons for me to use a message queue are:
I didn't have to reinvent the wheel. Designing even the simplest message queue using database requires at least a couple of hours of ...
- 3,498
6
votes
Event bus vs PubSub
For the "publisher/subscriber" pattern, subscriber components have to subscribe (or unsubscribe) to an event publisher. This means the life time of the event publishers is as least as long ...
- 203k
5
votes
Using flat files vs database/API as a transport between a frontend and backend
While the good 'ol save a file and copy it to a done dir is a staple of many communication layers esp. with older main frame systems and the like. The 'anti' guys do have a point; in that it has many ...
- 72.4k
5
votes
Solve Synchronization
As per the wikipedia page you linked to regarding message queues:
Message queues provide an asynchronous communications protocol,
meaning that the sender and receiver of the message do not need ...
- 11.7k
5
votes
Accepted
Should we be using pub/sub in our messaging stack?
I have written a similar system about a month ago, and just like you I've discovered that this is best solved by a reliable queue technology with a redrive.
While some might argue using SQL as your ...
- 384
5
votes
Accepted
How to deal with failing messages in DDD?
One way to deal with this is to use pull-based listeners instead of push-based. Each listener keeps track of its last read message and can request "messages since X". You could use either polling or a ...
- 4,361
5
votes
Accepted
Message Bus v Mediator pattern v In Memory Bus
A mediator pattern creates a flexible decoupled interface between two microservices. Messages are sent to and from each microservice without one necessarily having to know all the explicit workings ...
- 22.7k
5
votes
Communicating bulk data among microservices
I work on a system that has similar requirements - partners drop files that contain anywhere from 10 - 1,000,000 records to be processed. Some partners drop the files once a day, some once an hour, ...
- 286
5
votes
How to handle data inconsistency in microservice architecture?
I believe there's three typical ways you could handle this:
Guarantee delivery of your messages
Run a reconciliation process
Switch to a pulled events approach
A little detail on each approach...
...
- 213
5
votes
Accepted
Distributed message queue, propagating queue leader/follower information
The problem with the last option is that I would still need somewhere
to store queue metadata information like creation date.
Other than you may store other fields (like timestamp) in Metadata ...
- 3,162
4
votes
Message persistence in Message Queues
Consider a client that wishes to push a message to a queue. The queue manager would typically send a message acknowledgement to the client to signify that the message was successfully delivered to ...
- 26.4k
4
votes
Accepted
What is an appropriate pattern for handling a queue of network events in C++?
While processing the messages in the queue, you probably want to call a processing function that is specific to the most-derived type of the message you are looking at. The one place where that ...
- 73.2k
4
votes
Accepted
Best practice for delaying jobs
Your two main concerns here in high level design are essentially, the messaging from one or more software components to other software components, and flexible scheduling of jobs based on the age of a ...
- 26.4k
4
votes
How to implement "viewed: N times" functionality for an article?
I would suggest updating the counter on each view. As @amon suggested - that hard part is determining WHEN to update the counter.
An old adage - no premature optimization before its time. ...
- 141
4
votes
Accepted
How do I set up short-lived queues?
ActiveMQ does support this and it's a general feature of JMS. Here's a link pointing to a way of doing this.
I can't find the specific documentation at the moment but my recollection is that you ...
- 25.9k
Only top scored, non community-wiki answers of a minimum length are eligible