Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [message-queue]

A message queue is a service that receives events or messages from producers and provides them to consumers.

Filter by
Sorted by
Tagged with
0 votes
3 answers
81 views

Backend to client communication: WebSocket or Message Queue?

I have a hobby project, for simplicity imagine a poker game, consisting of a backend (Spring) and a client application (Flutter) where the client can join rooms in which he can take a seat. The client ...
mvi's user avatar
  • 9
2 votes
4 answers
87 views

Ensure concurrent processing based on a message key

I've been wondering about the following: Say you have an async message/task and you want that message only processed one at the time. for example: Only process one order for each customer at the time. ...
Joel Harkes's user avatar
0 votes
0 answers
35 views

What's the right ZMQ architecture for my scenario?

I have a service A that every once in a while needs to send a message to a cluster of services B1, B2 ... BN. Then all of these services need to receive that message reliably, and send a confirmation ...
codefast's user avatar
0 votes
2 answers
109 views

How to implement event payload isolation in an event driven architecture?

I have an event, let's say order.placed The payload consists of some general information about the order and 3 confidential fields. For example part of the message should only be visible to consumer A,...
Kamil Latosinski's user avatar
0 votes
1 answer
71 views

Design a sequential processing of records

About 10 to 15 records are processed per day with the time interval of 5 minutes between each record. System A inserts a record in DB and sends id of that record to active mq. System B Listener ...
sunleo's user avatar
  • 119
1 vote
0 answers
54 views

golang: pattern for handling message queues? Are named functions anti-idiomatic somehow?

Had a discussion today in how to implement services that work with messages coming in from event queues. We call these services processors. One of us argues for using several functions, while the ...
Albert Balbastre-Morte's user avatar
1 vote
0 answers
71 views

parent correlation id in message with rabbit queue and open telemetry

I have a rabbit queue that I use to communicate between applications. I have a chain of three message. App 1 generate and push a message in rabbit(message 1) App 2 catch the message and generate ...
user3401335's user avatar
3 votes
2 answers
228 views

Can multiple producers cause events with the same key to be enqueued out of order within a specific partition of a topic?

I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper ...
Shahriar Niloy's user avatar
0 votes
1 answer
86 views

Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?

I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
Richard's user avatar
  • 101
0 votes
1 answer
129 views

Generic message broker based on WCF

I apologize in advance for a long-read question, but i'm hardly stuck here and don`t how to ask for help using fewer words. I need to implement a middleware service which should be placed between two ...
Алексей К.'s user avatar
0 votes
0 answers
68 views

Event based update of In memory Hashmap in multiple pods of kubernetes

I have a service (S1) that is deployed on kubernetes having multiple replicas (pods). S1 requires some data from database (Cosmos Cassandra Api) for calculation. This data is constantly updated by ...
Lord Nick's user avatar
  • 101
1 vote
3 answers
183 views

Long-polling with message queues in a clustered environment

I have a system design question that I'm looking for some guidance on. I have two different systems that need to have a basic level of communication. This is abstracted via message queues. For example,...
user1597121's user avatar
1 vote
1 answer
68 views

Handling a single event as broadcast and round-robin depending on service

I have Service A that publishes an event to RabbitMQ. I have two instances of Service B that will use the event to write to its database. I have two instances of Service C that will send the event ...
DaveO's user avatar
  • 136
0 votes
0 answers
97 views

How to use an external event queue with inversion of control (IoC)

I'm working on an AspNetCore application, with a requirement to raise and handle certain events asynchronously using an external event queue. I'll use AWS services as examples here because that's what ...
brads3290's user avatar
  • 135
0 votes
1 answer
249 views

Is a message queue the right pattern to use for intra-module communication in a .net core application

Background Information We have a .Net core application that uses Hangfire (pretty version of cron) jobs to batch process different types of data. As a bogus example, we have one job that will run ...
dot's user avatar
  • 571
-1 votes
3 answers
345 views

Avoid race condition using RDBMS transaction

I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
Siddhartha Sadhukhan's user avatar
0 votes
2 answers
155 views

Complicated task scheduling architecture

So I want to make an application in which a user will hit an endpoint to save a job model to storage that includes some metadata to perform a long computation against which will be offloaded to a ...
Malik Brahimi's user avatar
2 votes
3 answers
208 views

Organize and maintain a lot of cron jobs

I am working on a fairly large project written in PHP (Yii2) in which we are increasingly using cron for background tasks, such as generating caches, reports, etc. We have started to move the hourly ...
MKMarek's user avatar
  • 29
1 vote
2 answers
4k views

Is Kafka an event bus or a publisher subscriber (pubsub)pattern?

Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
bodhihammer's user avatar
0 votes
2 answers
169 views

Is it a good/bad practice for a service to send events to itself via an external queue?

Imagine there is an Order entity that has a lot of states and conditional transitions between them. At the moment it is stored in a relational DB. There are multiple consumers of Orders that perform a ...
svz's user avatar
  • 307
0 votes
2 answers
153 views

How can I track the status of multiple messages in asynchronous processes?

Let's say I've a sendMessage endpoint. This endpoint looks at configured message destinations and then puts 1..N messages into a queue (e.g. {userId: 3, type: SMS, trackingID: X} and {userId: 3, type: ...
user avatar
0 votes
3 answers
103 views

Queue for tasks and notifications

I've been a heavy user of "batch jobs" to perform different kinds of logic in systems. Over the last couple of weeks, I've been thinking and reading about other approaches and I wanted to ...
Markus Knappen Johansson's user avatar
0 votes
1 answer
100 views

How to seamlessly interact with a message broker when the underlying system (SQS, RabbitMQ) can change depending on the environment?

Having a poor knowledge of the extended features of the available message brokers, I was wondering how to approach this. We have some environments where only RabbitMQ is available, others where ...
kant312's user avatar
  • 111
0 votes
2 answers
310 views

Message broker design pattern best practice [closed]

I'm designing the architecture of a platform to introduce a message broker in an existing data collection web application. This web application is currently used to upload data from excel files that ...
Giox's user avatar
  • 109
0 votes
1 answer
47 views

Which techniques to follow to make a fast and consistent message consumer that writes to a RDBMS [closed]

I am building a consumer application that reads messages from a queue and runs basic sanitization and deduplication and persists them to an RDBMS. The functional requirements are messages must not be ...
Sujal Mandal's user avatar
-1 votes
3 answers
262 views

How can i handle really long-term running tasks?

Im not dealing with a x minute task that can use an message queue and some workers to be done. Lets say that i have an cctv camera that capture 24/7, but can be stopped or started again via Web API ...
Vrrl's user avatar
  • 1
1 vote
2 answers
2k views

Implement message queue in API Gateway

Recently we have this client who asks for a API Gateway solution, in this case it’s Kong. Currently, they have 10 services (200 APIs) that are running on really legacy stuff (built with C++ and ...
Rex Low's user avatar
  • 119
1 vote
0 answers
80 views

How to have only one consumer handle a long running task at a given point?

I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items ...
Yos's user avatar
  • 157
0 votes
1 answer
54 views

Queue system with priority on Consumers

Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost. The system should ensure the messages are ...
Ryan's user avatar
  • 320
0 votes
1 answer
162 views

Bubbling errors upstream in async message-based services

Imagine a simple set up of an API and a 2nd service, where the API pushes some msgs to the message queue and the service pulls them and processes them. Now, if an error occurs while processing a msg, ...
Milkncookiez's user avatar
0 votes
0 answers
47 views

How to handle maintenance to your queuing server (e.g. rabbitMQ)

I'm new to queuing and looking into options that could be done operationally. I'm sorry if I have any conceptual misunderstandings related to queuing. I watched this video where the presenter made a ...
nanotek's user avatar
  • 301
6 votes
6 answers
464 views

Best practice to architect a system in which local users may override global definitions

Our travel industry client, operating across three continents and several countries, wants to standardize the process of choosing travel packages, air routes, hotels, and hundreds of other travel ...
Denis123's user avatar
1 vote
0 answers
271 views

Do we really need a message brocker in WebSocket system?

I'm engineering architecture of a new web based software. I've never worked on high-scale softwares before and I'm reading a lot about it. To increase client-side speed and reduce load on servers, and ...
vusra's user avatar
  • 19
0 votes
1 answer
415 views

How to optimize average rating calculation in a review system?

I'm thinking of a designing a review system (restaurant, hotel etc) where users can drop star reviews. Typically in a such a application, you can see the average rating of an entity along with all ...
Ahmed Sadman Muhib's user avatar
-1 votes
1 answer
423 views

Message queuing systems: sequential processing with dependencies between queues

I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
beater's user avatar
  • 107
-1 votes
1 answer
84 views

Any pattern for delaying processing of a (JMS) message until another task completes

Is there a good pattern in microservices systems for delaying the processing of a JMS message until some other task has been completed. Let's say, for example, a message is sent to a topic when a ...
BarrySW19's user avatar
  • 179
0 votes
1 answer
88 views

Expose api to third parties to send PDF files (invoices)

Our system needs to process PDF invoice files from generic external third parties. Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads. Even ...
Trace's user avatar
  • 185
2 votes
2 answers
1k views

Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?

We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
phen0menon's user avatar
0 votes
2 answers
188 views

Prevent missing updates using a message broker after creating a task

I'm trying to design a data updates mechanism in my micro-services architecture. For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
Sawel's user avatar
  • 109
3 votes
1 answer
713 views

Long-running compute-intensive tasks in APIs: background workers?

I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing. To give an impression of the compute requirements, ...
Korijn's user avatar
  • 139
0 votes
0 answers
37 views

Design a Program which Forwards Work Pieces to Corresponding Modules

I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
Conner's user avatar
  • 101
2 votes
2 answers
511 views

A service having to have both an API and listen to events - Is this an anti-pattern for microservices?

I have a use case to create a service that consume messages from a message queue, process them, store them in the DB, and expose the processed results via an API. Therefore, the service I'm going to ...
Deepal's user avatar
  • 129
2 votes
1 answer
4k views

Event bus vs PubSub

I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
Frankster's user avatar
  • 169
0 votes
2 answers
269 views

Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?

I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
Yos's user avatar
  • 157
1 vote
1 answer
705 views

What is the common practice to serve a REST API based request asynchronously?

I am building a microservice app where the client sends requests through REST APIs to the backend server, and the backend will communicate with other services using gRPC. I am aware that this is ...
prototype26's user avatar
2 votes
1 answer
1k views

Handling Errors in a Microservice architecture and asynchronous message queue

When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
JonasLevin's user avatar
0 votes
2 answers
398 views

Scaled services and order of events sent as messages (using a broker)

How would you go about making sure that "events" sent as messages to any sort of broker (Kafka, RabbitMQ etc) from multiple instances of the same service (load balanced) are actually added ...
Inx51's user avatar
  • 273
0 votes
1 answer
608 views

Removing successful cases from SQS batch. Is it wrong?

I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled. The most common problem with this kind of setup that I have seen is that, if ...
Javier Bullrich's user avatar
3 votes
0 answers
693 views

System Design: Chat application in a distributed system context

I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
Filipe Costa's user avatar
-1 votes
2 answers
211 views

How to keep authentication in user context in reactive programming?

I want to execute everything in a user context. Meaning that if the request comes from one user then the service has a token of only this user and can't execute anything for another user. Now the ...
user980828's user avatar

1
2 3 4 5 6