Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
221
questions
3
votes
3
answers
17k
views
Kafka, is it possible to get the message offset after producing?
While using Apache Kafka, is it possible to get the offset of the message that has been produced?
From the Kafka documentation page it says:
each message in the partition are assigned a unique ...
2
votes
1
answer
348
views
Design question on synchronization of two asynchronous data streams
I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
CommunityBot
- 1
0
votes
1
answer
88
views
Message queuing systems: sequential processing with dependencies between queues
I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
- 12.4k
0
votes
1
answer
194
views
Removing successful cases from SQS batch. Is it wrong?
I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled.
The most common problem with this kind of setup that I have seen is that, if ...
CommunityBot
- 1
0
votes
1
answer
59
views
Prevent missing updates using a message broker after creating a task
I'm trying to design a data updates mechanism in my micro-services architecture.
For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
CommunityBot
- 1
-1
votes
1
answer
47
views
Any pattern for delaying processing of a (JMS) message until another task completes
Is there a good pattern in microservices systems for delaying the processing of a JMS message until some other task has been completed. Let's say, for example, a message is sent to a topic when a ...
- 455
0
votes
1
answer
63
views
Expose api to third parties to send PDF files (invoices)
Our system needs to process PDF invoice files from generic external third parties.
Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads.
Even ...
- 125k
1
vote
2
answers
549
views
RabbitMQ Consumer Architecture
I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
CommunityBot
- 1
0
votes
1
answer
76
views
A service having to have both an API and listen to events - Is this an anti-pattern for microservices?
I have a use case to create a service that consume messages from a message queue, process them, store them in the DB, and expose the processed results via an API. Therefore, the service I'm going to ...
CommunityBot
- 1
1
vote
2
answers
176
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 10.5k
1
vote
1
answer
118
views
Long-running compute-intensive tasks in APIs: background workers?
I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.
To give an impression of the compute requirements, ...
- 17.9k
2
votes
3
answers
2k
views
Message Broker in Client-Server-Applications
I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ).
A simplified diagram of my ...
CommunityBot
- 1
0
votes
0
answers
35
views
Design a Program which Forwards Work Pieces to Corresponding Modules
I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
- 1
1
vote
2
answers
582
views
Long polling and message brokers
I was reading this article, and the author uses HTTP long polling to design a chat application.
I was wondering how does this compare with a dedicated Message Broker services like Apache Kafka, or ...
- 1
2
votes
1
answer
451
views
RabbitMQ: Topics and Persistence/Durability
Right now I'm using Direct Exchanges (i.e. one queue per "topic", pretty much pushing messages directly to a queue). This works OK, but I'd like the flexibility of working with Topic Exchanges, ...
CommunityBot
- 1
2
votes
1
answer
414
views
What are the different messaging patterns for multiple producers to single consumer through a distributed queue?
We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
CommunityBot
- 1
1
vote
1
answer
131
views
Alternative architecture for multi-instance, client-to-client message relay/broker
Research leads me to believe the most common architecture for a multi-instance (horizontally-scaling) message relay service (the typical example being an instant messenger/chat service) is to ...
CommunityBot
- 1
1
vote
1
answer
441
views
Event bus vs PubSub
I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
- 186k
1
vote
1
answer
345
views
Pattern to limit message queue processing based on external API throttling
I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
CommunityBot
- 1
0
votes
0
answers
66
views
Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?
I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
- 145
1
vote
1
answer
190
views
What is the common practice to serve a REST API based request asynchronously?
I am building a microservice app where the client sends requests through REST APIs to the backend server, and the backend will communicate with other services using gRPC. I am aware that this is ...
- 125k
0
votes
0
answers
42
views
design for the order processing system
We have REST endpoint that that works like this
Retrieve payload from request
Generated unique id for payload and order_status and enrichment_status to PENDING
Store payload in Redis
Perform request ...
- 203
-1
votes
2
answers
730
views
How to ensure persistence in an event-driven microservice architecture?
Right now we have a set of micro services with a client-facing gateway responsible for authentication and routing. MySQL database and RabbitMQ for the message queue. Internal services may queue up ...
- 224
-1
votes
1
answer
221
views
Handling multiple events in Event Driven Architecture
Let's say I have
Service A that publishes Message 1 to be processed by Consumer A
The same service A that publishes Message 2 to be processed by Consumer B
I need to make sure that consumer B ...
- 224
1
vote
1
answer
400
views
Handling Errors in a Microservice architecture and asynchronous message queue
When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
- 107k
0
votes
2
answers
94
views
Scaled services and order of events sent as messages (using a broker)
How would you go about making sure that "events" sent as messages to any sort of broker (Kafka, RabbitMQ etc) from multiple instances of the same service (load balanced) are actually added ...
- 139
2
votes
0
answers
230
views
System Design: Chat application in a distributed system context
I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
- 21
-1
votes
2
answers
94
views
How to keep authentication in user context in reactive programming?
I want to execute everything in a user context. Meaning that if the request comes from one user then the service has a token of only this user and can't execute anything for another user.
Now the ...
- 85
0
votes
1
answer
71
views
Interactive and Batch traffic in one service [closed]
I am designing a workflow, and am trying to avoid parallel deployments of the same service. Thus I am looking to have one service that handles both interactive and batch traffic. My main concern is ...
- 22.2k
0
votes
1
answer
123
views
What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?
I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic.
How can ...
- 364
1
vote
1
answer
85
views
Exposing redis to external clients
We are building a system that runs on our cloud and that needs information from our clients network that must not be exposed openly. We have concluded that the only way this could work, is if our ...
- 151
1
vote
0
answers
87
views
Retry long-running message processing in case of processing node failure
I have two types of microservices: Data Collector and Data Processor. They communicate with each other via Azure Service Bus Queue. Data Collector collects data from different sources and, once done, ...
- 149
1
vote
1
answer
1k
views
What is the advantage of POSIX queue or just normal data structure queue?
I'm programming with the embedded device running Linux, where memory is less, I just have 64MB flash only. I planned to use queues for the thread communication.
In which I came across with using ...
- 137
0
votes
2
answers
71
views
Do I need a Message processing service before RabbitMQ (or any other message broker)
I hope that's a proper place to ask my question.
I am wondering how is best to integrate RabbitMq (or any other message broker) into my project.
There are 2 options:
Simply collect all messages, ...
- 1,613
1
vote
2
answers
161
views
In Publish-subscribe pattern can a microservice publish message for itself
I am designing a somewhat complex flow in publish subscribe pattern using AWS SNS, SQS , Microservice and Lambda functions. To break the whole flow into multiple logical & smaller sub-flows I am ...
- 145k
0
votes
0
answers
41
views
Architectural design for sending large amount of analytics data from production servers to s3 without impacting request performance
Lets say we have a server getting upto 1000 requests per second, serving them at p99 of 20ms (strong business case for not increasing this latency). The server gc parameters have been carefully tuned ...
- 109
0
votes
0
answers
20
views
AWS Serverless Service to trigger an event when something doesn't happen
I'd like to use some sort of AWS serverless technology to trigger an event when a clickstream stops receiving clicks. Imagine a website visitor clicking around a site. Within our current solution ...
- 327
2
votes
1
answer
116
views
Implementing a persistent searchable "queue"
I'm receiving a lot of a data entries from different (IOT) devices. I need to temporarily and persistently store this data until it is polled by the matching clients, after which it will be deleted.
...
- 29
3
votes
0
answers
256
views
Implementation of projections in event-sourced system
I'm working on a application which uses event-sourcing and CQRS to define it's domain model.
Background
We have implemented projections to aggregate stream of all domain events into a read models used ...
- 35.6k
0
votes
1
answer
455
views
Replacing in-memory queue with messaging queue service
In our application (ASP.NET core) we have events that require computations and which results need to be persisted in a database (the objects are projects that need recalculations when a property ...
- 835
0
votes
2
answers
863
views
What is the Difference between Reactive Extensions and Message Queue?
I am trying to understand the difference between Reactive Extensions and Message Queue? Are they competing frameworks? Can they be used in conjuction? Started programming, and trying to understand ...
CommunityBot
- 1
0
votes
0
answers
265
views
High availability RabbitMQ using gRPC as transport layer?
I have an idea to use gRPC for all microservices communication for the sake of simplicity.
Now there is a service that needs high availability, So the idea is to use RabitMQ as a message receiver from ...
- 657
0
votes
1
answer
199
views
How can message queues improve scalability?
I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
1
vote
1
answer
114
views
Eventing: Determine if Set of Messages has been Processed
Given a Set of messages sent to a Queue, what are good ways to determine all of the messages have been processed?
Constraints:
Large number of messages
Other messages besides those in the Set will be ...
- 649
0
votes
0
answers
260
views
gRPC client - to - client communication between other Lan network
I have a question about gRPC communication.
I know gRPC has 4 models communication: unay, server-streaming, client-streaming and bi-direction. And the terms "client" and "server" ...
3
votes
2
answers
138
views
Best approach for inter-process task queues
I have an application made up of multiple processes/workers/services which need to send messages to each other that represent units of "enqueued tasks" to be done. I am trying to find the ...
- 786
2
votes
1
answer
114
views
How (whether to?) include Apache Spark in my Architecture
Brief overview of general data flow
The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic ...
- 216
4
votes
2
answers
6k
views
Message Bus v Mediator pattern v In Memory Bus
I am struggling to understand when to use each of these:
1) Message Bus: Used to send integration events between Microservices. For example, Microservice A could publish an integration event, which ...
- 171
6
votes
2
answers
400
views
Design question for handling large volumes of messages in multi-tenant queue
I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario.
Producer produces events based ...
- 193
-1
votes
1
answer
112
views
How to prevent races inside HTTP API?
You're writing the code for an API that accepts payloads from users and, before performing the actual write/update/delete in the db must validate the input for consistency and this is done by ...
- 1,708