All Questions
Tagged with queue message-queue
7 questions
2
votes
2
answers
1k
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 83
1
vote
1
answer
1k
views
Pattern to limit message queue processing based on external API throttling
I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
- 157
6
votes
2
answers
748
views
Design question for handling large volumes of messages in multi-tenant queue
I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario.
Producer produces events based ...
- 169
3
votes
3
answers
297
views
Dealing with data arriving at a different times
I have a load balanced service that needs two data inputs to correctly give a result.
One of these inputs comes from a queue, the other comes from user input.
Most of the time the data that comes ...
- 4,057
9
votes
2
answers
2k
views
Fair distributed task scheduling with RabbitMQ
Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
- 199
2
votes
3
answers
3k
views
Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?
My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
- 2,476
6
votes
3
answers
1k
views
Slow throughput - still worth using a dedicated message queue?
Here are some requirements for a queue:
Every few days add ~100k tasks with various priorities
Workers will pull tasks at typically less than 10 / second
Tasks need to be completed by ~2 unique ...
- 163