Questions tagged [message-queue]

A message queue is a service that receives events or messages from producers and provides them to consumers.

Filter by
Sorted by
Tagged with
3 votes
3 answers
17k views

Kafka, is it possible to get the message offset after producing?

While using Apache Kafka, is it possible to get the offset of the message that has been produced? From the Kafka documentation page it says: each message in the partition are assigned a unique ...
2 votes
1 answer
348 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0 votes
1 answer
88 views

Message queuing systems: sequential processing with dependencies between queues

I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
0 votes
1 answer
194 views

Removing successful cases from SQS batch. Is it wrong?

I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled. The most common problem with this kind of setup that I have seen is that, if ...
0 votes
1 answer
59 views

Prevent missing updates using a message broker after creating a task

I'm trying to design a data updates mechanism in my micro-services architecture. For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
-1 votes
1 answer
47 views

Any pattern for delaying processing of a (JMS) message until another task completes

Is there a good pattern in microservices systems for delaying the processing of a JMS message until some other task has been completed. Let's say, for example, a message is sent to a topic when a ...
0 votes
1 answer
63 views

Expose api to third parties to send PDF files (invoices)

Our system needs to process PDF invoice files from generic external third parties. Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads. Even ...
1 vote
2 answers
549 views

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
0 votes
1 answer
76 views

A service having to have both an API and listen to events - Is this an anti-pattern for microservices?

I have a use case to create a service that consume messages from a message queue, process them, store them in the DB, and expose the processed results via an API. Therefore, the service I'm going to ...
1 vote
2 answers
176 views

Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?

We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
1 vote
1 answer
118 views

Long-running compute-intensive tasks in APIs: background workers?

I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing. To give an impression of the compute requirements, ...
2 votes
3 answers
2k views

Message Broker in Client-Server-Applications

I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ). A simplified diagram of my ...
0 votes
0 answers
35 views

Design a Program which Forwards Work Pieces to Corresponding Modules

I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
1 vote
2 answers
582 views

Long polling and message brokers

I was reading this article, and the author uses HTTP long polling to design a chat application. I was wondering how does this compare with a dedicated Message Broker services like Apache Kafka, or ...
2 votes
1 answer
451 views

RabbitMQ: Topics and Persistence/Durability

Right now I'm using Direct Exchanges (i.e. one queue per "topic", pretty much pushing messages directly to a queue). This works OK, but I'd like the flexibility of working with Topic Exchanges, ...
2 votes
1 answer
414 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
1 vote
1 answer
131 views

Alternative architecture for multi-instance, client-to-client message relay/broker

Research leads me to believe the most common architecture for a multi-instance (horizontally-scaling) message relay service (the typical example being an instant messenger/chat service) is to ...
1 vote
1 answer
441 views

Event bus vs PubSub

I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
1 vote
1 answer
345 views

Pattern to limit message queue processing based on external API throttling

I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
0 votes
0 answers
66 views

Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?

I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
1 vote
1 answer
190 views

What is the common practice to serve a REST API based request asynchronously?

I am building a microservice app where the client sends requests through REST APIs to the backend server, and the backend will communicate with other services using gRPC. I am aware that this is ...
0 votes
0 answers
42 views

design for the order processing system

We have REST endpoint that that works like this Retrieve payload from request Generated unique id for payload and order_status and enrichment_status to PENDING Store payload in Redis Perform request ...
-1 votes
2 answers
730 views

How to ensure persistence in an event-driven microservice architecture?

Right now we have a set of micro services with a client-facing gateway responsible for authentication and routing. MySQL database and RabbitMQ for the message queue. Internal services may queue up ...
-1 votes
1 answer
221 views

Handling multiple events in Event Driven Architecture

Let's say I have Service A that publishes Message 1 to be processed by Consumer A The same service A that publishes Message 2 to be processed by Consumer B I need to make sure that consumer B ...
1 vote
1 answer
400 views

Handling Errors in a Microservice architecture and asynchronous message queue

When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
0 votes
2 answers
94 views

Scaled services and order of events sent as messages (using a broker)

How would you go about making sure that "events" sent as messages to any sort of broker (Kafka, RabbitMQ etc) from multiple instances of the same service (load balanced) are actually added ...
2 votes
0 answers
230 views

System Design: Chat application in a distributed system context

I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
-1 votes
2 answers
94 views

How to keep authentication in user context in reactive programming?

I want to execute everything in a user context. Meaning that if the request comes from one user then the service has a token of only this user and can't execute anything for another user. Now the ...
0 votes
1 answer
71 views

Interactive and Batch traffic in one service [closed]

I am designing a workflow, and am trying to avoid parallel deployments of the same service. Thus I am looking to have one service that handles both interactive and batch traffic. My main concern is ...
0 votes
1 answer
123 views

What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?

I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic. How can ...
1 vote
1 answer
85 views

Exposing redis to external clients

We are building a system that runs on our cloud and that needs information from our clients network that must not be exposed openly. We have concluded that the only way this could work, is if our ...
1 vote
0 answers
87 views

Retry long-running message processing in case of processing node failure

I have two types of microservices: Data Collector and Data Processor. They communicate with each other via Azure Service Bus Queue. Data Collector collects data from different sources and, once done, ...
1 vote
1 answer
1k views

What is the advantage of POSIX queue or just normal data structure queue?

I'm programming with the embedded device running Linux, where memory is less, I just have 64MB flash only. I planned to use queues for the thread communication. In which I came across with using ...
0 votes
2 answers
71 views

Do I need a Message processing service before RabbitMQ (or any other message broker)

I hope that's a proper place to ask my question. I am wondering how is best to integrate RabbitMq (or any other message broker) into my project. There are 2 options: Simply collect all messages, ...
1 vote
2 answers
161 views

In Publish-subscribe pattern can a microservice publish message for itself

I am designing a somewhat complex flow in publish subscribe pattern using AWS SNS, SQS , Microservice and Lambda functions. To break the whole flow into multiple logical & smaller sub-flows I am ...
0 votes
0 answers
41 views

Architectural design for sending large amount of analytics data from production servers to s3 without impacting request performance

Lets say we have a server getting upto 1000 requests per second, serving them at p99 of 20ms (strong business case for not increasing this latency). The server gc parameters have been carefully tuned ...
0 votes
0 answers
20 views

AWS Serverless Service to trigger an event when something doesn't happen

I'd like to use some sort of AWS serverless technology to trigger an event when a clickstream stops receiving clicks. Imagine a website visitor clicking around a site. Within our current solution ...
2 votes
1 answer
116 views

Implementing a persistent searchable "queue"

I'm receiving a lot of a data entries from different (IOT) devices. I need to temporarily and persistently store this data until it is polled by the matching clients, after which it will be deleted. ...
3 votes
0 answers
256 views

Implementation of projections in event-sourced system

I'm working on a application which uses event-sourcing and CQRS to define it's domain model. Background We have implemented projections to aggregate stream of all domain events into a read models used ...
0 votes
1 answer
455 views

Replacing in-memory queue with messaging queue service

In our application (ASP.NET core) we have events that require computations and which results need to be persisted in a database (the objects are projects that need recalculations when a property ...
0 votes
2 answers
863 views

What is the Difference between Reactive Extensions and Message Queue?

I am trying to understand the difference between Reactive Extensions and Message Queue? Are they competing frameworks? Can they be used in conjuction? Started programming, and trying to understand ...
0 votes
0 answers
265 views

High availability RabbitMQ using gRPC as transport layer?

I have an idea to use gRPC for all microservices communication for the sake of simplicity. Now there is a service that needs high availability, So the idea is to use RabitMQ as a message receiver from ...
0 votes
1 answer
199 views

How can message queues improve scalability?

I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
1 vote
1 answer
114 views

Eventing: Determine if Set of Messages has been Processed

Given a Set of messages sent to a Queue, what are good ways to determine all of the messages have been processed? Constraints: Large number of messages Other messages besides those in the Set will be ...
0 votes
0 answers
260 views

gRPC client - to - client communication between other Lan network

I have a question about gRPC communication. I know gRPC has 4 models communication: unay, server-streaming, client-streaming and bi-direction. And the terms "client" and "server" ...
3 votes
2 answers
138 views

Best approach for inter-process task queues

I have an application made up of multiple processes/workers/services which need to send messages to each other that represent units of "enqueued tasks" to be done. I am trying to find the ...
2 votes
1 answer
114 views

How (whether to?) include Apache Spark in my Architecture

Brief overview of general data flow The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic ...
4 votes
2 answers
6k views

Message Bus v Mediator pattern v In Memory Bus

I am struggling to understand when to use each of these: 1) Message Bus: Used to send integration events between Microservices. For example, Microservice A could publish an integration event, which ...
6 votes
2 answers
400 views

Design question for handling large volumes of messages in multi-tenant queue

I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based ...
-1 votes
1 answer
112 views

How to prevent races inside HTTP API?

You're writing the code for an API that accepts payloads from users and, before performing the actual write/update/delete in the db must validate the input for consistency and this is done by ...

1
2 3 4 5