Questions tagged [message-queue]

A message queue is a service that receives events or messages from producers and provides them to consumers.

Filter by
Sorted by
Tagged with
47 votes
2 answers
27k views

When to use Advanced Message Queuing Protocol like RabbitMQ? [closed]

Can somebody explain me in what use-cases I should consider using AMQP like e.g. RabbitMQ? What are the pros and cons?
user avatar
  • 825
40 votes
1 answer
37k views

Why database as queue so bad? [closed]

I've just read this article, and I'm confused. Let's imagine 1 webapp and 1 distinct application acting as "worker", both sharing the same database. Oh, I said "sharing"..but what does the article ...
user avatar
  • 3,768
32 votes
3 answers
33k views

How to implement a message queue over Redis?

Why Redis for queuing? I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...
user avatar
  • 2,182
27 votes
5 answers
17k views

Message Queue. Database vs Dedicated MQ

I am after advice regarding message queueing. We have requirements for "jobs" to be posted to a message queue. The original suggestion was just to use a SQL Server instance and process messages from ...
user avatar
26 votes
1 answer
20k views

Designing an scalable message queue architecture

I have recently started learning the nuances of scalable and enterprise computer architecture, and one of the central components is a messaging queue. In order to learn the most I can from any ...
user avatar
  • 561
23 votes
1 answer
14k views

What are the solutions to the Distributed Queue problem?

I am trying to learn more about the various ways that the problem of a Distributed Queue may be solved. So I would like know what products, services, implementations and research papers that are ...
user avatar
22 votes
1 answer
10k views

Does Akka obsolesce JMS/AMQP message brokers? [closed]

I spent the last week deep diving into the Akka docs and finally understand what actor systems are, and the problems that they solve. My understanding (and experience with) traditional JMS/AMQP ...
user avatar
  • 4,680
19 votes
4 answers
8k views

Using flat files vs database/API as a transport between a frontend and backend

I've got an application which has generated a rather heated discussion between a couple of the developers. Basically, it's split into a web layer and a backend layer. The web layer collects ...
user avatar
17 votes
2 answers
7k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
user avatar
  • 4,680
14 votes
3 answers
24k views

UML: Queue processor in a sequence diagram

Which is the appropriate way of representing a queue processor in a UML sequence diagram? I want to represent in the same diagram two systems that are loosely coupled through a queue. The first ...
user avatar
10 votes
2 answers
14k views

REST or a message queue in a multi-tier heterogeneous system?

I'm designing a REST API for a three-tier system like: Client application -> Front-end API cloud server -> user's home API server (Home). Home is a home device, and is supposed to maintain connection ...
user avatar
9 votes
4 answers
2k views

Handling changes in a event-driven microservice architecture

I'm doing an research-project where I'm researching the options to handle changes in an event-driven microservice architecture. So, let's say we got an application where we got four different ...
user avatar
  • 191
8 votes
2 answers
2k views

Product agnostic book on Message Queues? [closed]

Are there any recommendations for book(s) that are largely agnostic (with examples of MQ implementations is fine) on vendor but go to great details on the architecture, management, nomenclature, ...
user avatar
  • 3,937
8 votes
2 answers
729 views

Microservices - compensate service failure with queue

We are using some sort of microservices approach in our app (although it is not really consequently adhered to). When a service is down or throwing an exception, the approach is to put it into a ...
user avatar
8 votes
3 answers
4k views

Message Queues and Multiple Subscribers

My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at ...
user avatar
8 votes
5 answers
1k views

Message validation in async messaging-based services

I'm looking for information on the best approach to message validation in asynchronous-messaging-based services (i.e. services that pull messages from some sort of message queue or broker, rather than ...
user avatar
  • 1,045
7 votes
2 answers
2k views

Implementation of communication between packages (Java)

I'm making a project with 5 packages. The packages can communicate with each other by sending Messages to a central MessageQueue in the Main package and that MessageQueue takes care of all the ...
user avatar
  • 173
7 votes
1 answer
7k views

Do I actually need a message broker or are websockets enough?

The website I am building has a real-time messaging component. The backend is built with Flask and I have integrated Flask-SocketIO to handle Websocket connections when users are on the messaging page....
user avatar
  • 1,627
7 votes
2 answers
2k views

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
user avatar
7 votes
1 answer
7k views

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

Current Scenario Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
user avatar
6 votes
3 answers
7k views

Implementing a caching microservice by avoiding potential bottlenecks

The system I am working on employs quite a sophisticated caching/preloading mechanism for external third-parties, and since it is built on a microservice architecture, I would like to extract the ...
user avatar
  • 161
6 votes
3 answers
946 views

Slow throughput - still worth using a dedicated message queue?

Here are some requirements for a queue: Every few days add ~100k tasks with various priorities Workers will pull tasks at typically less than 10 / second Tasks need to be completed by ~2 unique ...
user avatar
  • 163
6 votes
2 answers
4k views

How do I document a communication protocol on top of message queues and channels?

I'm working on a large project at the moment and I'm trying to document the communication protocol that sits on top of a message queue middleware. The documentation is intended for another ...
user avatar
  • 3,903
6 votes
2 answers
4k views

Understanding the Single-Writer Principle

The article on the Single-Writer Principle of the Mechanical Sympathy blog explains how bad queues are (performance-wise), because they need to be able to receive messages from multiple produers, and ...
user avatar
  • 81
6 votes
2 answers
400 views

Design question for handling large volumes of messages in multi-tenant queue

I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based ...
user avatar
  • 169
6 votes
2 answers
727 views

Handling cases where the message queue is unreachable

We are using Apache Camel with ActiveMQ in a microservice architecture in a growing system of integration solutions. I was wondering how each service should react when they are unable to connect to ...
user avatar
  • 135
6 votes
1 answer
767 views

Best practice for ensuring name uniqueness/correctness in message queues in a microservice architecture/distributed system

I was wondering what a good solution would be for ensuring that queue-names are entered correctly and are only used by the correct applications in a large system which uses message queues to exchange ...
user avatar
  • 135
5 votes
3 answers
3k views

Web Service and Message Queue

We're looking at creating a web services/REST API layer that will be consumed by web and mobile clients. To make the solution more robust I was considering putting the commands from PUT, POST, and ...
user avatar
5 votes
1 answer
1k views

How to deal with failing messages in DDD?

We have a set of micro-services all built according to the Domain Driven Design (DDD). The micro-services communicate via Domain Events with each other (OrderSubmittedEvent, CustomerBilledEvent, ...). ...
user avatar
5 votes
2 answers
138 views

How to structure messages between a web server and a job server

For the SaaS infrastructure I'm building, we need to send transactional emails and run other long-running tasks on a separate job server. I can communicate via a queue, such as AWS' SQS or via a ...
user avatar
5 votes
1 answer
1k views

Message queue vs database for delayed tasks

I need to build a system that can handle a fairly high amount of delayed tasks (e.g. scheduled emails). For non-delayed tasks I would go for something like RabbitMQ. But, is it ok to let tasks ...
user avatar
  • 53
5 votes
1 answer
5k views

Is having a bi directional message queue a design smell?

I have an architecture with 1 backend server and multiple frontend servers. The frontend servers are connected in a bi-directional (web sockets) connection to clients so they can send messages to ...
user avatar
5 votes
2 answers
4k views

Synchronization of data across microservices

We have 2 to 3 dozen microservices that serve our customers. These services are deployed in a Kubernetes cluster, and they're only accessible to the outside world through 3 or 4 API gateways. We ...
user avatar
5 votes
1 answer
231 views

How to Recover from Inconsistent Job State without Database Polling

I'm working on scaling an application which is currently polling a mySQL database to send async jobs to a background processing system. A state machine is being used to keep track of the entities ...
user avatar
  • 153
5 votes
2 answers
1k views

How to handle thousands of messages to C socket?

I currently have a setup looking like this: __________________ _________ ________ ___________ | Front end server |----| Varnish |----| NodeJS |----| C-service | ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ...
user avatar
  • 151
4 votes
2 answers
6k views

Message Bus v Mediator pattern v In Memory Bus

I am struggling to understand when to use each of these: 1) Message Bus: Used to send integration events between Microservices. For example, Microservice A could publish an integration event, which ...
user avatar
  • 6,865
4 votes
2 answers
2k views

Where does Windows Workflow Foundation fit into the messaging oriented middleware architecture?

I've been trying to find information online about where WWF 4.5 fits with in an architecture based around an enterprise message queue using central broker(s) (RabbitMQ in particular). It seems like ...
user avatar
  • 143
4 votes
1 answer
3k views

AMQP routing-keys naming anti-patterns when using topics

I'm working on the specification for building a web services platform and AMQP is the messaging protocol used for the interaction between the components of this platform. I'm using topic exchanges ...
user avatar
  • 151
4 votes
2 answers
2k views

Communicating bulk data among microservices

Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication. My use case is to import data from upstream feed files(say 50k records) to our ...
user avatar
4 votes
3 answers
6k views

NoSQL Document Database as a Message Queue

I am considering using a NoSQL Document database as a messaging queue. Here is why: I want a client to post a message (some serialized object) to the server and not have to wait for a synchronous ...
user avatar
  • 879
4 votes
2 answers
3k views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
user avatar
  • 143
4 votes
1 answer
1k views

Message driven architecture and horizontal scaling

During working on a pet project in microservice event driven architecture, I've faced a problem of delivering responses to a client. Each microservice (API Gateway etc) has many replicas and has it's ...
user avatar
  • 431
4 votes
2 answers
739 views

Master-Slave Cluster - How to make sure the master is really dead for the slave to take over?

I have a in-house messaging system, similar to a message broker. We have one master message broker and one slave message broker. A message broker just receives messages and sends them to all nodes. ...
user avatar
4 votes
3 answers
2k views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
user avatar
  • 317
4 votes
1 answer
3k views

Architecture for real-time application, avoiding duplicate messages for queues with no active consumers

I am designing the architecture of an application which has your typical backend and frontend components. The frontend application must be semi real-time. For this requirement, the backend is pushing ...
user avatar
4 votes
1 answer
834 views

distributed application design - using message broker

Background I'm trying to design a layer/component in my distributed application that will communicate between component A and component B. Right now, this "communication" is accomplished by ...
user avatar
  • 479
4 votes
1 answer
2k views

How many Topics (queues) to practically use in an ESB

In all the Service Bus examples and tutorials I've seen, they cover publishing a single message to a single Topic that is picked up by a single listening Subscriber. Whilst this is good for a simple ...
user avatar
4 votes
1 answer
115 views

Message publishing interface when deferring until transaction is committed

Suppose I have the following interface used for publishing messages onto a message queue: interface IMessageProducer { void Publish<TMessage>(TMessage message); } Normally, an implementer ...
user avatar
4 votes
1 answer
2k views

Is it always the best way to go to queue emails instead of sending them real-time? [duplicate]

If you had to decide between queueing emails and sending them real-time, is it always the best way to go putting them in a queue and sending them one by one instead of instantly as soon as a user ...
user avatar
  • 235
4 votes
1 answer
573 views

Azure ServiceBus Queue: Consumer Application

I am running a .net MVC4 web application. I want to be able to hit an action on a controller, create an email and send it to my service bus queue to be processed (sent). What are my options as far ...
user avatar
  • 141

1
2 3 4 5