Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
252
questions
1
vote
2
answers
2k
views
Message Queue with multiple consumers locking synchronous on a field
We have a system where multiple consumers get messages out of a queue (currently implemented with a database). The messages have an order field. All messages of one particular order should be ...
1
vote
2
answers
6k
views
Queueing API requests
Is it a good practice at all?
E.g. we have an API layer which has to call some service when it receives a request, and the work which that service has to do must not be done concurrently. If there ...
- 35
4
votes
2
answers
3k
views
Communicating bulk data among microservices
Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication.
My use case is to import data from upstream feed files(say 50k records) to our ...
- 304
0
votes
3
answers
211
views
Keeping up logical order in a messaging environment
My setting is the following (Please assume that this points are entirely rock solid unchangeable, some for good reasons some: "just because"):
In a super scalable microservice environment i receive ...
- 456
26
votes
1
answer
20k
views
Designing an scalable message queue architecture
I have recently started learning the nuances of scalable and enterprise computer architecture, and one of the central components is a messaging queue. In order to learn the most I can from any ...
- 37.2k
0
votes
2
answers
522
views
Moving single tenant application with queue to multi tenants web application
I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
- 2,897
2
votes
3
answers
3k
views
Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?
My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
- 2,446
9
votes
1
answer
11k
views
Do I actually need a message broker or are websockets enough?
The website I am building has a real-time messaging component. The backend is built with Flask and I have integrated Flask-SocketIO to handle Websocket connections when users are on the messaging page....
CommunityBot
- 1
5
votes
2
answers
5k
views
Synchronization of data across microservices
We have 2 to 3 dozen microservices that serve our customers.
These services are deployed in a Kubernetes cluster, and they're only accessible to the outside world through 3 or 4 API gateways.
We ...
- 1,444
0
votes
2
answers
191
views
Possible microservice based design
Here are the design considerations for a software.
The querying of data should be as fast as possible (reader <1s, writer <2.5s possibly since it also take time to reach the client end web or ...
- 499
0
votes
0
answers
18
views
Technique to match a message by criteria to node in the cluster
I have global message queue with multiple different producers, and a cluster with N nodes of consumers. Each consumer receives a message and may decide to process it or to put it back to the queue if ...
- 126
1
vote
1
answer
117
views
Best practice for multi-application communication and events?
Essentially, I have a web application and a console application. The web application allows users to configure what will be processed, which is just a big information processor. From the web UI, a ...
- 2,066
0
votes
1
answer
85
views
How to implement adaptive routing?
I need to send jobs to worker queues according to resource usage and other metrics like how many jobs have been processed and how successful they were.
There is also the need to “weight” these ...
CommunityBot
- 1
9
votes
5
answers
2k
views
Message validation in async messaging-based services
I'm looking for information on the best approach to message validation in asynchronous-messaging-based services (i.e. services that pull messages from some sort of message queue or broker, rather than ...
8
votes
3
answers
6k
views
Message Queues and Multiple Subscribers
My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at ...
- 48.6k
2
votes
0
answers
375
views
Data pipeline architecture: airflow triggered by message broker
Let us say we have:
a web app with a Postgres DB that produces data over time,
another DB optimized for analytics that we would like to populate over time.
My goal is to build and monitor an ETL ...
- 151
0
votes
2
answers
119
views
How to check if the command is execuable when it delivers through asynchronous message queue?
The aggregate root's state is the product of the commands it receives. So different order of receiving the commands produce different aggregate root states.
How can I check domain rules based on the ...
- 1,591
2
votes
0
answers
621
views
Connect a web-app to to the back-end message queue
We are currently looking into dividing our back-end in multiple services using a message queue (RabbitMQ). Having a message queue suddenly gives us the possibility to update the web-app freely, but do ...
- 129
1
vote
1
answer
861
views
How can an event subscriber catch up with missed events?
At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time.
For example, a service is running and subscribes to events on ...
- 11.7k
1
vote
2
answers
324
views
Acceptable to use synchronous call to another microservice for time-sensitve state change?
Say there are two microservices (example is simplified)
PickupRequestService: lists pick-up requests of passengers
DriverService: drivers use to accept pickup requests
On a completely decoupled ...
- 147k
-1
votes
1
answer
77
views
Bringing data from 30 APIs based on data came from the APIs
I have to write server (.net core) that is going to read data from 30 different remote APIs. I suppose to run some sort of decision trees, if I find something in one API goes to another API, else some ...
- 27.3k
1
vote
1
answer
93
views
Approach to handle network breakdowns in application dealing with database?
I have an application that writes data to database. In case of network breakdowns, the connection to database gets lost & my db queries freezes the whole application.
So, in order to make it ...
- 207k
0
votes
2
answers
599
views
Should db commit and message-queue produce be one transaction?
I am making a microservice based architecture and I'm actively seeking antipatterns and trying to avoid them. In multiple scenarios, I have a database commit followed by a message that is produced to ...
- 2,051
9
votes
2
answers
2k
views
Fair distributed task scheduling with RabbitMQ
Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
CommunityBot
- 1
0
votes
1
answer
222
views
Which message pattern should I use?
I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
- 326
3
votes
3
answers
287
views
Dealing with data arriving at a different times
I have a load balanced service that needs two data inputs to correctly give a result.
One of these inputs comes from a queue, the other comes from user input.
Most of the time the data that comes ...
- 4,048
0
votes
0
answers
749
views
Using signalR for streaming alongside a microservice architecture
I have a cluster of micro services with an exposed search API serving a client app. I want to "stream" in real time the aggregated results of a complex search. The search will be kicked of by the ...
- 1
0
votes
1
answer
1k
views
Scheduling operations in a micro service
I'm trying to realize a service that executes operations at a certain time, those operations are specified by another service by messaging, here's the scenario:
The service A receives a request to ...
- 134k
1
vote
0
answers
203
views
Reporting progress on queued tasks
This might seem as a very basic and maybe a vague question but I'd really appreciate advice and experience sharing on this:
Given you have a service, which processes tasks in a queue, e.g. because ...
- 279
3
votes
1
answer
1k
views
AWS SQS FIFO Limit Workaround
According to the AWS feature page for SQS:
FIFO queues support up to 300 messages per second
Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.
I'm ...
CommunityBot
- 1
0
votes
1
answer
58
views
Horizontally Scalable Streaming of SQL Query Results
I have a service that runs queries on a database and gets (potentially) large number of rows. I also have services that can export a given set of rows to csv, xlsx or pdf etc. I have separate export ...
- 111
1
vote
1
answer
557
views
Queuing emails in Laravel when using an api driver such as Mailgun?
In Laravel (for the record I'm using v5.6) there is an option to queue emails for background processing.
I have a scheduled job to send out daily email alerts to users. Currently the job drops the ...
- 37.2k
2
votes
1
answer
449
views
MQ Architecture: How to handle old messages in the queue and releasing a consumer/producer upgrade
So for example if I have a producer that sends a message
{id: 1}
to a consumer that expects that json object. And there's a ton of messages sitting in the queue waiting to be processed. I'm also ...
- 7,126
10
votes
2
answers
15k
views
REST or a message queue in a multi-tier heterogeneous system?
I'm designing a REST API for a three-tier system like: Client application -> Front-end API cloud server -> user's home API server (Home).
Home is a home device, and is supposed to maintain connection ...
- 1,249
30
votes
5
answers
20k
views
Message Queue. Database vs Dedicated MQ
I am after advice regarding message queueing. We have requirements for "jobs" to be posted to a message queue.
The original suggestion was just to use a SQL Server instance and process messages from ...
- 121
2
votes
1
answer
49
views
Decomposition by "role" or by "usage" - is that a thing?
I have a mail_queue table that is written to by web requests then worked on by a worker to send the mails. I've now had a moment of enlightenment re. what I was not comfortable with in the design.
...
- 2,764
0
votes
1
answer
390
views
How an async response is usually getting back to the clent in typical reactive highload architecture?
A typical highload reactive architecture is:
(Q means queue)
And usually arrows are from client to DB at such sketches. My question is how the response is going back? The same queues and other stuff?...
- 134k
0
votes
1
answer
3k
views
Processing only once the same message produced by two producers
If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed?
Is the only way to have an ...
- 77.2k
40
votes
1
answer
41k
views
Why database as queue so bad? [closed]
I've just read this article, and I'm confused.
Let's imagine 1 webapp and 1 distinct application acting as "worker", both sharing the same database.
Oh, I said "sharing"..but what does the article ...
- 382
-3
votes
1
answer
36
views
Disable recieving messages from queue remotly
I have a CRUD like service which have REST HTTP API. This service have two instances - only one recieving request at time. If first instance starts throwing exceptions or slowing down we change load ...
- 75.7k
2
votes
2
answers
2k
views
When consuming an api, what is a good way to deal with their request limits?
My app is consuming a third party api. One of the requirements of this api is that my app cannot send more than 20 requests per second. Because of the nature of this app, and because my user base is ...
- 311
0
votes
1
answer
2k
views
How to load test Publisher/Subscribing queue?
I am trying to load test a queue that works on a pub sub pattern and can't figure out how response time is to be calculate and the TPS(transactions per second).
Problem statement :
The pub sub ...
- 154
2
votes
1
answer
422
views
Architecture pattern for temporary message queue subscriptions with websocket clients
We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
- 29
23
votes
1
answer
10k
views
Does Akka obsolesce JMS/AMQP message brokers? [closed]
I spent the last week deep diving into the Akka docs and finally understand what actor systems are, and the problems that they solve.
My understanding (and experience with) traditional JMS/AMQP ...
- 280
4
votes
1
answer
1k
views
Should we be using pub/sub in our messaging stack?
Basically this question is of the variety "Can I check that we're not about to do something dumb?"
We're setting up a new system that needs to guarantee delivery of emails.
We have an ...
CommunityBot
- 1
1
vote
1
answer
300
views
How to architect master/slave store/messages and de-duplication?
To clarify, as I'm not entirely sure how to summarise in the title.
I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc).
My redis cluster has a master and a couple ...
- 307
2
votes
1
answer
273
views
What is the better place to create the Binding and Queue on RPC communication: Consumer or Producer?
I was talking with part of my team about the responsibility for who own the Binding and Queue creation. We are using Spring framework.
In async communication they agree that the responsability to ...
- 2,446
1
vote
2
answers
1k
views
Architecture for Queuing based technology in a Micro-Service, Multi-Tenant Environment
Hopefully the title is not painfully obscure, but as it indicates, I'm really looking for advice on my architecture proposal, as the current architecture is subject to concurrency issues, performance ...
CommunityBot
- 1
0
votes
1
answer
141
views
Where should I store statistics about sent emails?
my application sends a batch of custom emails every day using mailgun (an email API service). I have a producer worker scheduled to run daily, and all does is query the database for the unique ...
- 269
5
votes
1
answer
4k
views
Architecture for real-time application, avoiding duplicate messages for queues with no active consumers
I am designing the architecture of an application which has your typical backend and frontend components. The frontend application must be semi real-time. For this requirement, the backend is pushing ...
CommunityBot
- 1