Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
250
questions
-1
votes
1
answer
38
views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.
System A inserts a record in DB and sends id of that record to active mq.
System B Listener ...
- 107
0
votes
0
answers
30
views
golang: pattern for handling message queues? Are named functions anti-idiomatic somehow?
Had a discussion today in how to implement services that work with messages coming in from event queues. We call these services processors. One of us argues for using several functions, while the ...
0
votes
0
answers
27
views
parent correlation id in message with rabbit queue and open telemetry
I have a rabbit queue that I use to communicate between applications.
I have a chain of three message.
App 1 generate and push a message in rabbit(message 1)
App 2 catch the message and generate ...
- 277
3
votes
1
answer
122
views
Can multiple producers cause events with the same key to be enqueued out of order within a specific partition of a topic?
I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper ...
0
votes
1
answer
72
views
Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?
I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
- 101
0
votes
1
answer
84
views
Generic message broker based on WCF
I apologize in advance for a long-read question, but i'm hardly stuck here and don`t how to ask for help using fewer words.
I need to implement a middleware service which should be placed between two ...
0
votes
0
answers
49
views
Event based update of In memory Hashmap in multiple pods of kubernetes
I have a service (S1) that is deployed on kubernetes having multiple replicas (pods).
S1 requires some data from database (Cosmos Cassandra Api) for calculation.
This data is constantly updated by ...
- 101
0
votes
3
answers
94
views
Long-polling with message queues in a clustered environment
I have a system design question that I'm looking for some guidance on. I have two different systems that need to have a basic level of communication. This is abstracted via message queues.
For example,...
- 101
1
vote
1
answer
52
views
Handling a single event as broadcast and round-robin depending on service
I have Service A that publishes an event to RabbitMQ. I have two instances of Service B that will use the event to write to its database. I have two instances of Service C that will send the event ...
- 136
0
votes
0
answers
89
views
How to use an external event queue with inversion of control (IoC)
I'm working on an AspNetCore application, with a requirement to raise and handle certain events asynchronously using an external event queue. I'll use AWS services as examples here because that's what ...
- 135
0
votes
1
answer
184
views
Is a message queue the right pattern to use for intra-module communication in a .net core application
Background Information
We have a .Net core application that uses Hangfire (pretty version of cron) jobs to batch process different types of data.
As a bogus example, we have one job that will run ...
- 541
-1
votes
3
answers
186
views
Avoid race condition using RDBMS transaction
I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
0
votes
2
answers
133
views
Complicated task scheduling architecture
So I want to make an application in which a user will hit an endpoint to save a job model to storage that includes some metadata to perform a long computation against which will be offloaded to a ...
- 101
2
votes
3
answers
165
views
Organize and maintain a lot of cron jobs
I am working on a fairly large project written in PHP (Yii2) in which we are increasingly using cron for background tasks, such as generating caches, reports, etc.
We have started to move the hourly ...
- 29
0
votes
2
answers
2k
views
Is Kafka an event bus or a publisher subscriber (pubsub)pattern?
Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
- 101
0
votes
2
answers
122
views
Is it a good/bad practice for a service to send events to itself via an external queue?
Imagine there is an Order entity that has a lot of states and conditional transitions between them. At the moment it is stored in a relational DB.
There are multiple consumers of Orders that perform a ...
- 307
0
votes
2
answers
117
views
How can I track the status of multiple messages in asynchronous processes?
Let's say I've a sendMessage endpoint. This endpoint looks at configured message destinations and then puts 1..N messages into a queue (e.g. {userId: 3, type: SMS, trackingID: X} and {userId: 3, type: ...
- 149
0
votes
3
answers
90
views
Queue for tasks and notifications
I've been a heavy user of "batch jobs" to perform different kinds of logic in systems. Over the last couple of weeks, I've been thinking and reading about other approaches and I wanted to ...
0
votes
1
answer
77
views
How to seamlessly interact with a message broker when the underlying system (SQS, RabbitMQ) can change depending on the environment?
Having a poor knowledge of the extended features of the available message brokers, I was wondering how to approach this.
We have some environments where only RabbitMQ is available, others where ...
- 111
0
votes
2
answers
166
views
Message broker design pattern best practice [closed]
I'm designing the architecture of a platform to introduce a message broker in an existing data collection web application.
This web application is currently used to upload data from excel files that ...
- 109
0
votes
1
answer
45
views
Which techniques to follow to make a fast and consistent message consumer that writes to a RDBMS [closed]
I am building a consumer application that reads messages from a queue and runs basic sanitization and deduplication and persists them to an RDBMS.
The functional requirements are
messages must not be ...
- 131
-1
votes
3
answers
219
views
How can i handle really long-term running tasks?
Im not dealing with a x minute task that can use an message queue and some workers to be done. Lets say that i have an cctv camera that capture 24/7, but can be stopped or started again via Web API ...
- 1
1
vote
2
answers
949
views
Implement message queue in API Gateway
Recently we have this client who asks for a API Gateway solution, in this case it’s Kong. Currently, they have 10 services (200 APIs) that are running on really legacy stuff (built with C++ and ...
- 119
1
vote
0
answers
66
views
How to have only one consumer handle a long running task at a given point?
I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items ...
- 155
0
votes
1
answer
54
views
Queue system with priority on Consumers
Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost.
The system should ensure the messages are ...
- 320
0
votes
1
answer
139
views
Bubbling errors upstream in async message-based services
Imagine a simple set up of an API and a 2nd service, where the API pushes some msgs to the message queue and the service pulls them and processes them.
Now, if an error occurs while processing a msg, ...
- 171
0
votes
0
answers
41
views
How to handle maintenance to your queuing server (e.g. rabbitMQ)
I'm new to queuing and looking into options that could be done operationally. I'm sorry if I have any conceptual misunderstandings related to queuing.
I watched this video where the presenter made a ...
- 301
4
votes
6
answers
411
views
Best practice to architect a system in which local users may override global definitions
Our travel industry client, operating across three continents and several countries, wants to standardize the process of choosing travel packages, air routes, hotels, and hundreds of other travel ...
- 57
0
votes
0
answers
170
views
Do we really need a message brocker in WebSocket system?
I'm engineering architecture of a new web based software.
I've never worked on high-scale softwares before and I'm reading a lot about it.
To increase client-side speed and reduce load on servers, and ...
- 9
0
votes
1
answer
314
views
How to optimize average rating calculation in a review system?
I'm thinking of a designing a review system (restaurant, hotel etc) where users can drop star reviews. Typically in a such a application, you can see the average rating of an entity along with all ...
-1
votes
1
answer
244
views
Message queuing systems: sequential processing with dependencies between queues
I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
- 107
-1
votes
1
answer
76
views
Any pattern for delaying processing of a (JMS) message until another task completes
Is there a good pattern in microservices systems for delaying the processing of a JMS message until some other task has been completed. Let's say, for example, a message is sent to a topic when a ...
- 179
0
votes
1
answer
80
views
Expose api to third parties to send PDF files (invoices)
Our system needs to process PDF invoice files from generic external third parties.
Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads.
Even ...
- 185
2
votes
2
answers
848
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 73
0
votes
1
answer
134
views
Prevent missing updates using a message broker after creating a task
I'm trying to design a data updates mechanism in my micro-services architecture.
For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
- 109
3
votes
1
answer
587
views
Long-running compute-intensive tasks in APIs: background workers?
I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.
To give an impression of the compute requirements, ...
- 139
0
votes
0
answers
36
views
Design a Program which Forwards Work Pieces to Corresponding Modules
I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
- 1
2
votes
2
answers
348
views
A service having to have both an API and listen to events - Is this an anti-pattern for microservices?
I have a use case to create a service that consume messages from a message queue, process them, store them in the DB, and expose the processed results via an API. Therefore, the service I'm going to ...
- 129
1
vote
1
answer
3k
views
Event bus vs PubSub
I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
- 159
0
votes
1
answer
200
views
Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?
I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
- 155
1
vote
1
answer
551
views
What is the common practice to serve a REST API based request asynchronously?
I am building a microservice app where the client sends requests through REST APIs to the backend server, and the backend will communicate with other services using gRPC. I am aware that this is ...
- 23
1
vote
1
answer
1k
views
Handling Errors in a Microservice architecture and asynchronous message queue
When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
- 129
0
votes
2
answers
307
views
Scaled services and order of events sent as messages (using a broker)
How would you go about making sure that "events" sent as messages to any sort of broker (Kafka, RabbitMQ etc) from multiple instances of the same service (load balanced) are actually added ...
- 261
0
votes
1
answer
551
views
Removing successful cases from SQS batch. Is it wrong?
I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled.
The most common problem with this kind of setup that I have seen is that, if ...
- 119
3
votes
0
answers
629
views
System Design: Chat application in a distributed system context
I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
- 31
-1
votes
2
answers
175
views
How to keep authentication in user context in reactive programming?
I want to execute everything in a user context. Meaning that if the request comes from one user then the service has a token of only this user and can't execute anything for another user.
Now the ...
- 364
0
votes
1
answer
83
views
Interactive and Batch traffic in one service [closed]
I am designing a workflow, and am trying to avoid parallel deployments of the same service. Thus I am looking to have one service that handles both interactive and batch traffic. My main concern is ...
- 109
0
votes
1
answer
219
views
What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?
I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic.
How can ...
- 119
1
vote
1
answer
321
views
Exposing redis to external clients
We are building a system that runs on our cloud and that needs information from our clients network that must not be exposed openly. We have concluded that the only way this could work, is if our ...
- 13
1
vote
0
answers
149
views
Retry long-running message processing in case of processing node failure
I have two types of microservices: Data Collector and Data Processor. They communicate with each other via Azure Service Bus Queue. Data Collector collects data from different sources and, once done, ...
- 159