Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
257
questions
-2
votes
1
answer
57
views
Messaging queque for batch inserting into database? [closed]
Currently I have a multi threaded Java Spring Boot application that inserts/updates database records. Currently one database transaction is carried out for each SQL insert and update which is very ...
- 105
1
vote
1
answer
112
views
Background thread processing vs queue based processing for relatively short tasks (max 30-40 seconds)
I need suggestion / recommendation on the design approaches mentioned below.
UseCase: I have a usecase where a client uses my system to generate some recommendations. Now, these recommendations are ...
- 19
3
votes
3
answers
174
views
Handling rate limits / delays in consumers without affecting performance of other operations
I have a producer that generates a batch job that consists of multiple operations (approx. 100 - 10000). These operations can be processed in any order, ideally as fast as possible.
The processing of ...
- 141
0
votes
1
answer
71
views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
7
votes
5
answers
380
views
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
We have this architecture:
queue -> message processor (horizontal scaling) -> RDBMS
Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
- 229
0
votes
3
answers
428
views
Backend to client communication: WebSocket or Message Queue?
I have a hobby project, for simplicity imagine a poker game, consisting of a backend (Spring) and a client application (Flutter) where the client can join rooms in which he can take a seat. The client ...
- 9
2
votes
4
answers
144
views
Ensure concurrent processing based on a message key
I've been wondering about the following: Say you have an async message/task and you want that message only processed one at the time. for example: Only process one order for each customer at the time. ...
- 174
0
votes
2
answers
175
views
How to implement event payload isolation in an event driven architecture?
I have an event, let's say order.placed The payload consists of some general information about the order and 3 confidential fields. For example part of the message should only be visible to consumer A,...
- 129
0
votes
1
answer
80
views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.
System A inserts a record in DB and sends id of that record to active mq.
System B Listener ...
- 119
1
vote
0
answers
77
views
golang: pattern for handling message queues? Are named functions anti-idiomatic somehow?
Had a discussion today in how to implement services that work with messages coming in from event queues. We call these services processors. One of us argues for using several functions, while the ...
1
vote
0
answers
84
views
parent correlation id in message with rabbit queue and open telemetry
I have a rabbit queue that I use to communicate between applications.
I have a chain of three message.
App 1 generate and push a message in rabbit(message 1)
App 2 catch the message and generate ...
- 297
3
votes
2
answers
299
views
Can multiple producers cause events with the same key to be enqueued out of order within a specific partition of a topic?
I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper ...
0
votes
1
answer
103
views
Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?
I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
- 101
0
votes
1
answer
139
views
Generic message broker based on WCF
I apologize in advance for a long-read question, but i'm hardly stuck here and don`t how to ask for help using fewer words.
I need to implement a middleware service which should be placed between two ...
0
votes
0
answers
86
views
Event based update of In memory Hashmap in multiple pods of kubernetes
I have a service (S1) that is deployed on kubernetes having multiple replicas (pods).
S1 requires some data from database (Cosmos Cassandra Api) for calculation.
This data is constantly updated by ...
- 101
1
vote
3
answers
241
views
Long-polling with message queues in a clustered environment
I have a system design question that I'm looking for some guidance on. I have two different systems that need to have a basic level of communication. This is abstracted via message queues.
For example,...
- 111
1
vote
1
answer
77
views
Handling a single event as broadcast and round-robin depending on service
I have Service A that publishes an event to RabbitMQ. I have two instances of Service B that will use the event to write to its database. I have two instances of Service C that will send the event ...
- 136
0
votes
0
answers
111
views
How to use an external event queue with inversion of control (IoC)
I'm working on an AspNetCore application, with a requirement to raise and handle certain events asynchronously using an external event queue. I'll use AWS services as examples here because that's what ...
- 135
0
votes
1
answer
334
views
Is a message queue the right pattern to use for intra-module communication in a .net core application
Background Information
We have a .Net core application that uses Hangfire (pretty version of cron) jobs to batch process different types of data.
As a bogus example, we have one job that will run ...
- 571
-1
votes
3
answers
418
views
Avoid race condition using RDBMS transaction
I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
0
votes
2
answers
190
views
Complicated task scheduling architecture
So I want to make an application in which a user will hit an endpoint to save a job model to storage that includes some metadata to perform a long computation against which will be offloaded to a ...
- 101
3
votes
4
answers
290
views
Organize and maintain a lot of cron jobs
I am working on a fairly large project written in PHP (Yii2) in which we are increasingly using cron for background tasks, such as generating caches, reports, etc.
We have started to move the hourly ...
- 39
1
vote
2
answers
5k
views
Is Kafka an event bus or a publisher subscriber (pubsub)pattern?
Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
- 121
0
votes
2
answers
197
views
Is it a good/bad practice for a service to send events to itself via an external queue?
Imagine there is an Order entity that has a lot of states and conditional transitions between them. At the moment it is stored in a relational DB.
There are multiple consumers of Orders that perform a ...
- 307
0
votes
2
answers
181
views
How can I track the status of multiple messages in asynchronous processes?
Let's say I've a sendMessage endpoint. This endpoint looks at configured message destinations and then puts 1..N messages into a queue (e.g. {userId: 3, type: SMS, trackingID: X} and {userId: 3, type: ...
user374484
0
votes
3
answers
121
views
Queue for tasks and notifications
I've been a heavy user of "batch jobs" to perform different kinds of logic in systems. Over the last couple of weeks, I've been thinking and reading about other approaches and I wanted to ...
0
votes
1
answer
114
views
How to seamlessly interact with a message broker when the underlying system (SQS, RabbitMQ) can change depending on the environment?
Having a poor knowledge of the extended features of the available message brokers, I was wondering how to approach this.
We have some environments where only RabbitMQ is available, others where ...
- 111
0
votes
2
answers
468
views
Message broker design pattern best practice [closed]
I'm designing the architecture of a platform to introduce a message broker in an existing data collection web application.
This web application is currently used to upload data from excel files that ...
- 109
0
votes
1
answer
50
views
Which techniques to follow to make a fast and consistent message consumer that writes to a RDBMS [closed]
I am building a consumer application that reads messages from a queue and runs basic sanitization and deduplication and persists them to an RDBMS.
The functional requirements are
messages must not be ...
- 131
-1
votes
3
answers
286
views
How can i handle really long-term running tasks?
Im not dealing with a x minute task that can use an message queue and some workers to be done. Lets say that i have an cctv camera that capture 24/7, but can be stopped or started again via Web API ...
- 1
1
vote
2
answers
2k
views
Implement message queue in API Gateway
Recently we have this client who asks for a API Gateway solution, in this case it’s Kong. Currently, they have 10 services (200 APIs) that are running on really legacy stuff (built with C++ and ...
- 119
1
vote
0
answers
85
views
How to have only one consumer handle a long running task at a given point?
I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items ...
- 157
0
votes
1
answer
55
views
Queue system with priority on Consumers
Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost.
The system should ensure the messages are ...
- 320
0
votes
1
answer
182
views
Bubbling errors upstream in async message-based services
Imagine a simple set up of an API and a 2nd service, where the API pushes some msgs to the message queue and the service pulls them and processes them.
Now, if an error occurs while processing a msg, ...
- 171
0
votes
0
answers
49
views
How to handle maintenance to your queuing server (e.g. rabbitMQ)
I'm new to queuing and looking into options that could be done operationally. I'm sorry if I have any conceptual misunderstandings related to queuing.
I watched this video where the presenter made a ...
- 301
6
votes
6
answers
479
views
Best practice to architect a system in which local users may override global definitions
Our travel industry client, operating across three continents and several countries, wants to standardize the process of choosing travel packages, air routes, hotels, and hundreds of other travel ...
- 77
1
vote
0
answers
325
views
Do we really need a message brocker in WebSocket system?
I'm engineering architecture of a new web based software.
I've never worked on high-scale softwares before and I'm reading a lot about it.
To increase client-side speed and reduce load on servers, and ...
- 19
0
votes
1
answer
455
views
How to optimize average rating calculation in a review system?
I'm thinking of a designing a review system (restaurant, hotel etc) where users can drop star reviews. Typically in a such a application, you can see the average rating of an entity along with all ...
-1
votes
1
answer
531
views
Message queuing systems: sequential processing with dependencies between queues
I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
- 107
-1
votes
1
answer
86
views
Any pattern for delaying processing of a (JMS) message until another task completes
Is there a good pattern in microservices systems for delaying the processing of a JMS message until some other task has been completed. Let's say, for example, a message is sent to a topic when a ...
- 179
0
votes
1
answer
100
views
Expose api to third parties to send PDF files (invoices)
Our system needs to process PDF invoice files from generic external third parties.
Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads.
Even ...
- 185
2
votes
2
answers
1k
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 73
0
votes
2
answers
202
views
Prevent missing updates using a message broker after creating a task
I'm trying to design a data updates mechanism in my micro-services architecture.
For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
- 109
3
votes
1
answer
834
views
Long-running compute-intensive tasks in APIs: background workers?
I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.
To give an impression of the compute requirements, ...
- 139
0
votes
0
answers
38
views
Design a Program which Forwards Work Pieces to Corresponding Modules
I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
- 101
2
votes
2
answers
701
views
A service having to have both an API and listen to events - Is this an anti-pattern for microservices?
I have a use case to create a service that consume messages from a message queue, process them, store them in the DB, and expose the processed results via an API. Therefore, the service I'm going to ...
- 129
2
votes
1
answer
5k
views
Event bus vs PubSub
I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
- 169
0
votes
2
answers
286
views
Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?
I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
- 157
1
vote
1
answer
792
views
What is the common practice to serve a REST API based request asynchronously?
I am building a microservice app where the client sends requests through REST APIs to the backend server, and the backend will communicate with other services using gRPC. I am aware that this is ...
- 23
2
votes
1
answer
1k
views
Handling Errors in a Microservice architecture and asynchronous message queue
When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
- 139