Questions tagged [message-queue]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
2answers
64 views

Moving single tenant application with queue to multi tenants web application

I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
0
votes
0answers
10 views

Distribute processing load across multiple nodes of same Web Application

We have an application deployed across multiple nodes. There is a scheduled job inside this application that would be run on any one of the nodes. When the job is triggered it is supposed to read ...
1
vote
1answer
64 views

Alternative architecture for multi-instance, client-to-client message relay/broker

Research leads me to believe the most common architecture for a multi-instance (horizontally-scaling) message relay service (the typical example being an instant messenger/chat service) is to ...
3
votes
1answer
90 views

Do I actually need a message broker or are websockets enough?

The website I am building has a real-time messaging component. The backend is built with Flask and I have integrated Flask-SocketIO to handle Websocket connections when users are on the messaging page....
0
votes
2answers
74 views

Possible microservice based design

Here are the design considerations for a software. The querying of data should be as fast as possible (reader <1s, writer <2.5s possibly since it also take time to reach the client end web or ...
0
votes
0answers
12 views

Technique to match a message by criteria to node in the cluster

I have global message queue with multiple different producers, and a cluster with N nodes of consumers. Each consumer receives a message and may decide to process it or to put it back to the queue if ...
1
vote
1answer
121 views

What is the Difference between Reactive Extensions and Message Queue?

I am trying to understand the difference between Reactive Extensions and Message Queue? Are they competing frameworks? Can they be used in conjuction? Started programming, and trying to understand ...
1
vote
1answer
71 views

Best practice for multi-application communication and events?

Essentially, I have a web application and a console application. The web application allows users to configure what will be processed, which is just a big information processor. From the web UI, a ...
1
vote
1answer
129 views

Message Queue with multiple consumers locking synchronous on a field

We have a system where multiple consumers get messages out of a queue (currently implemented with a database). The messages have an order field. All messages of one particular order should be ...
0
votes
0answers
22 views

Are message identifiers and offsets the same thing in Kafka?

I am reading up on Kafka and encountered the following on one of the tutorials. The producer sends the following to the broker - unique message identifier, timestamp and a binary payload. The ...
2
votes
0answers
72 views

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL ...
2
votes
1answer
119 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0
votes
2answers
75 views

How to check if the command is execuable when it delivers through asynchronous message queue?

The aggregate root's state is the product of the commands it receives. So different order of receiving the commands produce different aggregate root states. How can I check domain rules based on the ...
2
votes
0answers
70 views

Connect a web-app to to the back-end message queue

We are currently looking into dividing our back-end in multiple services using a message queue (RabbitMQ). Having a message queue suddenly gives us the possibility to update the web-app freely, but do ...
0
votes
1answer
96 views

How can an event subscriber catch up with missed events?

At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time. For example, a service is running and subscribes to events on ...
1
vote
1answer
142 views

Reduce duplicates in outbox pattern in event driven systems

Trying to implement the outbox pattern for an event driven system. The outbox pattern in a nutshell is a way to ensure system events are sent to the event log/queue/bus at least once (using the term ...
1
vote
2answers
78 views

Acceptable to use synchronous call to another microservice for time-sensitve state change?

Say there are two microservices (example is simplified) PickupRequestService: lists pick-up requests of passengers DriverService: drivers use to accept pickup requests On a completely decoupled ...
0
votes
1answer
70 views

Bringing data from 30 APIs based on data came from the APIs

I have to write server (.net core) that is going to read data from 30 different remote APIs. I suppose to run some sort of decision trees, if I find something in one API goes to another API, else some ...
0
votes
1answer
264 views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
1
vote
1answer
134 views

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
1
vote
1answer
81 views

Approach to handle network breakdowns in application dealing with database?

I have an application that writes data to database. In case of network breakdowns, the connection to database gets lost & my db queries freezes the whole application. So, in order to make it ...
0
votes
2answers
145 views

Should db commit and message-queue produce be one transaction?

I am making a microservice based architecture and I'm actively seeking antipatterns and trying to avoid them. In multiple scenarios, I have a database commit followed by a message that is produced to ...
0
votes
1answer
227 views

How to ensure persistence in an event-driven microservice architecture?

Right now we have a set of micro services with a client-facing gateway responsible for authentication and routing. MySQL database and RabbitMQ for the message queue. Internal services may queue up ...
4
votes
2answers
858 views

Synchronization of data across microservices

We have 2 to 3 dozen microservices that serve our customers. These services are deployed in a Kubernetes cluster, and they're only accessible to the outside world through 3 or 4 API gateways. We ...
0
votes
1answer
78 views

How to implement adaptive routing?

I need to send jobs to worker queues according to resource usage and other metrics like how many jobs have been processed and how successful they were. There is also the need to “weight” these ...
0
votes
1answer
158 views

Which message pattern should I use?

I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
0
votes
0answers
206 views

Using signalR for streaming alongside a microservice architecture

I have a cluster of micro services with an exposed search API serving a client app. I want to "stream" in real time the aggregated results of a complex search. The search will be kicked of by the ...
3
votes
3answers
249 views

Dealing with data arriving at a different times

I have a load balanced service that needs two data inputs to correctly give a result. One of these inputs comes from a queue, the other comes from user input. Most of the time the data that comes ...
0
votes
1answer
161 views

Scheduling operations in a micro service

I'm trying to realize a service that executes operations at a certain time, those operations are specified by another service by messaging, here's the scenario: The service A receives a request to ...
0
votes
0answers
56 views

Reporting progress on queued tasks

This might seem as a very basic and maybe a vague question but I'd really appreciate advice and experience sharing on this: Given you have a service, which processes tasks in a queue, e.g. because ...
0
votes
0answers
16 views

SOAP message exchange issue

We have a system that receive payment information for invoices from the bank system. It works like that: Invoice is created in our system Payer pays for the invoice from system of Bank Bank connects ...
1
vote
1answer
896 views

Queueing API requests

Is it a good practice at all? E.g. we have an API layer which has to call some service when it receives a request, and the work which that service has to do must not be done concurrently. If there ...
1
vote
2answers
396 views

Message Broker in Client-Server-Applications

I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ). A simplified diagram of my ...
2
votes
1answer
735 views

AWS SQS FIFO Limit Workaround

According to the AWS feature page for SQS: FIFO queues support up to 300 messages per second Standard queues support a nearly unlimited number of transactions per second (TPS) per API action. ...
0
votes
1answer
47 views

Horizontally Scalable Streaming of SQL Query Results

I have a service that runs queries on a database and gets (potentially) large number of rows. I also have services that can export a given set of rows to csv, xlsx or pdf etc. I have separate export ...
1
vote
1answer
280 views

Queuing emails in Laravel when using an api driver such as Mailgun?

In Laravel (for the record I'm using v5.6) there is an option to queue emails for background processing. I have a scheduled job to send out daily email alerts to users. Currently the job drops the ...
2
votes
1answer
203 views

MQ Architecture: How to handle old messages in the queue and releasing a consumer/producer upgrade

So for example if I have a producer that sends a message {id: 1} to a consumer that expects that json object. And there's a ton of messages sitting in the queue waiting to be processed. I'm also ...
2
votes
1answer
42 views

Decomposition by “role” or by “usage” - is that a thing?

I have a mail_queue table that is written to by web requests then worked on by a worker to send the mails. I've now had a moment of enlightenment re. what I was not comfortable with in the design. ...
0
votes
1answer
1k views

Message Bus v Mediator pattern v In Memory Bus

I am struggling to understand when to use each of these: 1) Message Bus: Used to send integration events between Microservices. For example, Microservice A could publish an integration event, which ...
0
votes
1answer
188 views

How an async response is usually getting back to the clent in typical reactive highload architecture?

A typical highload reactive architecture is: (Q means queue) And usually arrows are from client to DB at such sketches. My question is how the response is going back? The same queues and other stuff?...
0
votes
1answer
1k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
3
votes
2answers
908 views

How to handle concurrent updates for the same record in a distributed system

I'm trying to figure out the best solution for the below. Any help would be great. So basically I have a service (that can be scaled horizontally), which listens on a queue. Every message received ...
5
votes
3answers
856 views

Message Queues and Multiple Subscribers

My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at ...
-3
votes
1answer
33 views

Disable recieving messages from queue remotly

I have a CRUD like service which have REST HTTP API. This service have two instances - only one recieving request at time. If first instance starts throwing exceptions or slowing down we change load ...
2
votes
2answers
406 views

When consuming an api, what is a good way to deal with their request limits?

My app is consuming a third party api. One of the requirements of this api is that my app cannot send more than 20 requests per second. Because of the nature of this app, and because my user base is ...
0
votes
1answer
958 views

How to load test Publisher/Subscribing queue?

I am trying to load test a queue that works on a pub sub pattern and can't figure out how response time is to be calculate and the TPS(transactions per second). Problem statement : The pub sub ...
2
votes
1answer
250 views

Architecture pattern for temporary message queue subscriptions with websocket clients

We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
6
votes
2answers
803 views

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
1
vote
3answers
1k views

Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?

My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
1
vote
1answer
223 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...