Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
221
questions
1
vote
1
answer
38
views
optimal way of dealing with subscriptions
I have a system in which some data is inserted into database. Whenever that data is inserted subscribers should be notified about that (it doesn't have to happen just after inserting, but there is a ...
- 11
1
vote
1
answer
194
views
Designing Kafka topics for secured event driven job scheduling system
Consider the following
Group A
Job A {
Depends on Job B of Group A
Run User -> User1
}
Job B {
Depends on Job C and Job D of Group A
Run User -> User2
}
Job C {
Depends ...
- 119
0
votes
1
answer
69
views
Process AMPQ messages both reliably and fast
I am new to RabbitMQ and herewith I want to make sure that I am not missing out on some advanced RabbitMQ feature or pattern I am not aware of.
I need to develop a reliable system that processes a ...
- 139
0
votes
1
answer
153
views
Integrating with real-time data from multiple devices with accelerometer
I am designing a MVP for a simple gamification system for a trampoline park(s).
External company is providing bracelets for customers, then collecting this data in their own web app, where they ...
- 67
4
votes
2
answers
2k
views
Communicating bulk data among microservices
Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication.
My use case is to import data from upstream feed files(say 50k records) to our ...
- 141
0
votes
3
answers
152
views
Keeping up logical order in a messaging environment
My setting is the following (Please assume that this points are entirely rock solid unchangeable, some for good reasons some: "just because"):
In a super scalable microservice environment i receive ...
- 207
0
votes
2
answers
401
views
Moving single tenant application with queue to multi tenants web application
I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
- 121
1
vote
1
answer
131
views
Alternative architecture for multi-instance, client-to-client message relay/broker
Research leads me to believe the most common architecture for a multi-instance (horizontally-scaling) message relay service (the typical example being an instant messenger/chat service) is to ...
7
votes
1
answer
7k
views
Do I actually need a message broker or are websockets enough?
The website I am building has a real-time messaging component. The backend is built with Flask and I have integrated Flask-SocketIO to handle Websocket connections when users are on the messaging page....
- 1,627
0
votes
2
answers
168
views
Possible microservice based design
Here are the design considerations for a software.
The querying of data should be as fast as possible (reader <1s, writer <2.5s possibly since it also take time to reach the client end web or ...
- 271
0
votes
0
answers
15
views
Technique to match a message by criteria to node in the cluster
I have global message queue with multiple different producers, and a cluster with N nodes of consumers. Each consumer receives a message and may decide to process it or to put it back to the queue if ...
- 126
0
votes
2
answers
862
views
What is the Difference between Reactive Extensions and Message Queue?
I am trying to understand the difference between Reactive Extensions and Message Queue? Are they competing frameworks? Can they be used in conjuction? Started programming, and trying to understand ...
1
vote
1
answer
89
views
Best practice for multi-application communication and events?
Essentially, I have a web application and a console application. The web application allows users to configure what will be processed, which is just a big information processor. From the web UI, a ...
- 127
1
vote
2
answers
1k
views
Message Queue with multiple consumers locking synchronous on a field
We have a system where multiple consumers get messages out of a queue (currently implemented with a database). The messages have an order field. All messages of one particular order should be ...
2
votes
0
answers
281
views
Data pipeline architecture: airflow triggered by message broker
Let us say we have:
a web app with a Postgres DB that produces data over time,
another DB optimized for analytics that we would like to populate over time.
My goal is to build and monitor an ETL ...
- 151
2
votes
1
answer
347
views
Design question on synchronization of two asynchronous data streams
I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0
votes
2
answers
96
views
How to check if the command is execuable when it delivers through asynchronous message queue?
The aggregate root's state is the product of the commands it receives. So different order of receiving the commands produce different aggregate root states.
How can I check domain rules based on the ...
- 209
2
votes
0
answers
309
views
Connect a web-app to to the back-end message queue
We are currently looking into dividing our back-end in multiple services using a message queue (RabbitMQ). Having a message queue suddenly gives us the possibility to update the web-app freely, but do ...
- 129
0
votes
1
answer
494
views
How can an event subscriber catch up with missed events?
At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time.
For example, a service is running and subscribes to events on ...
- 385
1
vote
1
answer
466
views
Reduce duplicates in outbox pattern in event driven systems
Trying to implement the outbox pattern for an event driven system. The outbox pattern in a nutshell is a way to ensure system events are sent to the event log/queue/bus at least once (using the term ...
- 313
1
vote
2
answers
220
views
Acceptable to use synchronous call to another microservice for time-sensitve state change?
Say there are two microservices (example is simplified)
PickupRequestService: lists pick-up requests of passengers
DriverService: drivers use to accept pickup requests
On a completely decoupled ...
- 313
-1
votes
1
answer
77
views
Bringing data from 30 APIs based on data came from the APIs
I have to write server (.net core) that is going to read data from 30 different remote APIs. I suppose to run some sort of decision trees, if I find something in one API goes to another API, else some ...
- 185
4
votes
2
answers
3k
views
Architecture for message processing with scheduling, at scale
I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
- 143
1
vote
2
answers
548
views
RabbitMQ Consumer Architecture
I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
- 239
1
vote
1
answer
86
views
Approach to handle network breakdowns in application dealing with database?
I have an application that writes data to database. In case of network breakdowns, the connection to database gets lost & my db queries freezes the whole application.
So, in order to make it ...
- 111
0
votes
2
answers
452
views
Should db commit and message-queue produce be one transaction?
I am making a microservice based architecture and I'm actively seeking antipatterns and trying to avoid them. In multiple scenarios, I have a database commit followed by a message that is produced to ...
- 115
-1
votes
2
answers
730
views
How to ensure persistence in an event-driven microservice architecture?
Right now we have a set of micro services with a client-facing gateway responsible for authentication and routing. MySQL database and RabbitMQ for the message queue. Internal services may queue up ...
- 285
5
votes
2
answers
4k
views
Synchronization of data across microservices
We have 2 to 3 dozen microservices that serve our customers.
These services are deployed in a Kubernetes cluster, and they're only accessible to the outside world through 3 or 4 API gateways.
We ...
- 1,356
0
votes
1
answer
82
views
How to implement adaptive routing?
I need to send jobs to worker queues according to resource usage and other metrics like how many jobs have been processed and how successful they were.
There is also the need to “weight” these ...
- 11
0
votes
1
answer
203
views
Which message pattern should I use?
I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
- 11
0
votes
0
answers
611
views
Using signalR for streaming alongside a microservice architecture
I have a cluster of micro services with an exposed search API serving a client app. I want to "stream" in real time the aggregated results of a complex search. The search will be kicked of by the ...
- 1
3
votes
3
answers
264
views
Dealing with data arriving at a different times
I have a load balanced service that needs two data inputs to correctly give a result.
One of these inputs comes from a queue, the other comes from user input.
Most of the time the data that comes ...
- 3,978
0
votes
1
answer
694
views
Scheduling operations in a micro service
I'm trying to realize a service that executes operations at a certain time, those operations are specified by another service by messaging, here's the scenario:
The service A receives a request to ...
- 133
1
vote
0
answers
120
views
Reporting progress on queued tasks
This might seem as a very basic and maybe a vague question but I'd really appreciate advice and experience sharing on this:
Given you have a service, which processes tasks in a queue, e.g. because ...
- 279
1
vote
2
answers
4k
views
Queueing API requests
Is it a good practice at all?
E.g. we have an API layer which has to call some service when it receives a request, and the work which that service has to do must not be done concurrently. If there ...
- 279
2
votes
3
answers
2k
views
Message Broker in Client-Server-Applications
I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ).
A simplified diagram of my ...
- 141
3
votes
1
answer
1k
views
AWS SQS FIFO Limit Workaround
According to the AWS feature page for SQS:
FIFO queues support up to 300 messages per second
Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.
I'm ...
- 179
0
votes
1
answer
51
views
Horizontally Scalable Streaming of SQL Query Results
I have a service that runs queries on a database and gets (potentially) large number of rows. I also have services that can export a given set of rows to csv, xlsx or pdf etc. I have separate export ...
- 179
1
vote
1
answer
479
views
Queuing emails in Laravel when using an api driver such as Mailgun?
In Laravel (for the record I'm using v5.6) there is an option to queue emails for background processing.
I have a scheduled job to send out daily email alerts to users. Currently the job drops the ...
- 291
2
votes
1
answer
350
views
MQ Architecture: How to handle old messages in the queue and releasing a consumer/producer upgrade
So for example if I have a producer that sends a message
{id: 1}
to a consumer that expects that json object. And there's a ton of messages sitting in the queue waiting to be processed. I'm also ...
- 123
2
votes
1
answer
44
views
Decomposition by "role" or by "usage" - is that a thing?
I have a mail_queue table that is written to by web requests then worked on by a worker to send the mails. I've now had a moment of enlightenment re. what I was not comfortable with in the design.
...
- 121
4
votes
2
answers
6k
views
Message Bus v Mediator pattern v In Memory Bus
I am struggling to understand when to use each of these:
1) Message Bus: Used to send integration events between Microservices. For example, Microservice A could publish an integration event, which ...
- 6,865
0
votes
1
answer
369
views
How an async response is usually getting back to the clent in typical reactive highload architecture?
A typical highload reactive architecture is:
(Q means queue)
And usually arrows are from client to DB at such sketches. My question is how the response is going back? The same queues and other stuff?...
0
votes
1
answer
2k
views
Processing only once the same message produced by two producers
If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed?
Is the only way to have an ...
- 33
3
votes
2
answers
3k
views
How to handle concurrent updates for the same record in a distributed system
I'm trying to figure out the best solution for the below. Any help would be great.
So basically I have a service (that can be scaled horizontally), which listens on a queue. Every message received ...
- 47
8
votes
3
answers
4k
views
Message Queues and Multiple Subscribers
My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at ...
- 540
-3
votes
1
answer
36
views
Disable recieving messages from queue remotly
I have a CRUD like service which have REST HTTP API. This service have two instances - only one recieving request at time. If first instance starts throwing exceptions or slowing down we change load ...
- 101
2
votes
2
answers
1k
views
When consuming an api, what is a good way to deal with their request limits?
My app is consuming a third party api. One of the requirements of this api is that my app cannot send more than 20 requests per second. Because of the nature of this app, and because my user base is ...
- 159
0
votes
1
answer
2k
views
How to load test Publisher/Subscribing queue?
I am trying to load test a queue that works on a pub sub pattern and can't figure out how response time is to be calculate and the TPS(transactions per second).
Problem statement :
The pub sub ...
- 103
2
votes
1
answer
357
views
Architecture pattern for temporary message queue subscriptions with websocket clients
We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
- 29