Questions tagged [message-queue]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
4
votes
2answers
70 views

Design question for handling large volumes of messages in multi-tenant queue

I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based ...
-1
votes
1answer
103 views

How to prevent races inside HTTP API?

You're writing the code for an API that accepts payloads from users and, before performing the actual write/update/delete in the db must validate the input for consistency and this is done by ...
0
votes
0answers
19 views

Architecture - For cloud services like GCP Cloud Run, should a restful API also subscribe to a message queue and how

In a bit of a quandary. I am using Cloud Run on GCP for backend services, and it will auto-scale based on HTTP load, potentially scaling down to 0 if there is no traffic for awhile. I have a product ...
1
vote
1answer
132 views

Distributed message queue, propagating queue leader/follower information

I'm designing a distributed queue message queue and I'm not sure how to get the leader information to another component. This is my current design: I have the following components: FrontEnd: request ...
1
vote
2answers
89 views

How to make consumer idempotent when publisher publishes “at least once”?

I'm trying to implement microservices and I'm still super new to the subject, but, I was able to answer most of questions. However, one of the questions that I can't come up with a good solution to is ...
0
votes
0answers
40 views

Broker queue and topic naming conventions

I'm in the process of adding a message broker (ActiveMQ Artemis) to a system currently and while I have my own notions on how I can divide up the namespace, it seems like a good time to get opinions ...
0
votes
0answers
34 views

Scheduled messages pushed from database to API

I'm building a system with a dataflow, as described below. It is reasonable, or can it be improved, especially I would like comments on the job that moves messages from the database to the queue. ...
0
votes
3answers
224 views

How to handle data inconsistency in microservice architecture?

Let's assume we have two services - user and auth with a message broker in between. user service handles CRUD action on user entity and auth service handles authentication. When a user is created/...
1
vote
2answers
72 views

Service bus and strict message order - how can I handle failed messages?

I am considering using Azure Service Bus, however my question applies to queues in general. I have a requirement to send messages in a certain order. Say that the publisher sends a message to the ...
0
votes
0answers
23 views

What is the recommended way to start a long-running containerized analysis in R from NodeJS?

I have a frontend through which users can submit analysis requests, which are inserted into a database. These analysis requests are received in a NodeJS server using Postgres notify/listen. Using ...
0
votes
0answers
40 views

Integrating HTTP / Webhooks with Message Queues

I'm working at a project which integrates several Applications mostly SaaS Applications. The SaaS solutions have all the possibilities to hook into the internal event system with webhooks. The ...
0
votes
0answers
28 views

How to process a long list of data from database with Message-driven bean due to the asynchronous requirement properly

There is a task in our JavaEE project: a user in the frontend page A can click a button to request to get a zip with the processed data. As the process of the request in the backend needs plenty of ...
2
votes
1answer
128 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
0
votes
0answers
58 views

Is queue is the right choice for server request retries

I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. ...
0
votes
0answers
9 views

Scale spring-batch instances executed from within a web container

I am re-designing our current ETL for file ingestion where spring-batch containers will be deployed on Kubernetes. Each file type is processed by a different job, which will be triggered on demand via ...
0
votes
0answers
22 views

Kafka with log compaction best approach

I am publishing entities with entity-id (message-key) + version in kafka. I decided to use log compaction to ensure that atleast 1 version of the entity is available at all times and for that to be ...
0
votes
0answers
16 views

Messaging - Responsibility of Consumer vs Publisher for Active-Active failover

Say there is an external client-DC1/DC2, posting requests to a service in DC1 and DC2 respectively. Now, if DC1 is down, should the external client-DC1 responsible for routing the request message to ...
0
votes
1answer
61 views

Question about the best way to architect a small webapp

So I'm trying to create a simple website where I run a computationally intensive task on the backend based on input submitted by a form. For reference: my goal is not too dissimilar from something ...
1
vote
0answers
51 views

Using a message queue or pub / sub solution for connecting frontend to backend instead of REST APIs

I own a dell r620 server at home and want to make more use of it. Typically its my staging environment as my internet is currently ipv6 with dslite so I can’t port forward. But I can easily call and ...
1
vote
1answer
31 views

optimal way of dealing with subscriptions

I have a system in which some data is inserted into database. Whenever that data is inserted subscribers should be notified about that (it doesn't have to happen just after inserting, but there is a ...
1
vote
1answer
50 views

Designing Kafka topics for secured event driven job scheduling system

Consider the following Group A Job A { Depends on Job B of Group A Run User -> User1 } Job B { Depends on Job C and Job D of Group A Run User -> User2 } Job C { Depends ...
0
votes
0answers
29 views

Porting a SYSV IPC multiprocess architecture to Windows

I'm looking at porting a Unix-based multi-process architecture server application. The app uses SYS V IPC shared memory and semaphores. It has a main message queue in shared memory, with semaphores ...
0
votes
0answers
21 views

Can a common ActiveMQ service be shared in a dev environment without messages being consumed between developers?

Our dev environment is currently sharing an ActiveMQ service hosted on AWS. This however resulted in problems where dev A's service produce a message that will trigger dev B's localhost consumer. Is ...
0
votes
1answer
57 views

Process AMPQ messages both reliably and fast

I am new to RabbitMQ and herewith I want to make sure that I am not missing out on some advanced RabbitMQ feature or pattern I am not aware of. I need to develop a reliable system that processes a ...
-1
votes
0answers
84 views

Difference in AMPS and AMQP

I recently across an acronym AMPS(Asyncronous Message Processing System( by 60EastTechnologies. Since it is not a standard like AMQP(Asynchronous Message Queuing Protocol) or JMS(Java Messaging System)...
0
votes
0answers
242 views

Why control the minimum and maximum concurrent consumers on an AMQP queue?

I'm trying here to understand why some frameworks introduce the control over the number of consumers on a AMQP queue. For example, Spring AMQP introduces this feature on the 1.3.0 version, with the ...
0
votes
1answer
145 views

Integrating with real-time data from multiple devices with accelerometer

I am designing a MVP for a simple gamification system for a trampoline park(s). External company is providing bracelets for customers, then collecting this data in their own web app, where they ...
4
votes
2answers
427 views

Communicating bulk data among microservices

Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication. My use case is to import data from upstream feed files(say 50k records) to our ...
0
votes
3answers
97 views

Keeping up logical order in a messaging environment

My setting is the following (Please assume that this points are entirely rock solid unchangeable, some for good reasons some: "just because"): In a super scalable microservice environment i receive ...
0
votes
2answers
185 views

Moving single tenant application with queue to multi tenants web application

I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
1
vote
1answer
90 views

Alternative architecture for multi-instance, client-to-client message relay/broker

Research leads me to believe the most common architecture for a multi-instance (horizontally-scaling) message relay service (the typical example being an instant messenger/chat service) is to ...
5
votes
1answer
2k views

Do I actually need a message broker or are websockets enough?

The website I am building has a real-time messaging component. The backend is built with Flask and I have integrated Flask-SocketIO to handle Websocket connections when users are on the messaging page....
0
votes
2answers
110 views

Possible microservice based design

Here are the design considerations for a software. The querying of data should be as fast as possible (reader <1s, writer <2.5s possibly since it also take time to reach the client end web or ...
0
votes
0answers
13 views

Technique to match a message by criteria to node in the cluster

I have global message queue with multiple different producers, and a cluster with N nodes of consumers. Each consumer receives a message and may decide to process it or to put it back to the queue if ...
-1
votes
2answers
468 views

What is the Difference between Reactive Extensions and Message Queue?

I am trying to understand the difference between Reactive Extensions and Message Queue? Are they competing frameworks? Can they be used in conjuction? Started programming, and trying to understand ...
1
vote
1answer
77 views

Best practice for multi-application communication and events?

Essentially, I have a web application and a console application. The web application allows users to configure what will be processed, which is just a big information processor. From the web UI, a ...
1
vote
2answers
477 views

Message Queue with multiple consumers locking synchronous on a field

We have a system where multiple consumers get messages out of a queue (currently implemented with a database). The messages have an order field. All messages of one particular order should be ...
2
votes
0answers
194 views

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL ...
3
votes
1answer
201 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0
votes
2answers
76 views

How to check if the command is execuable when it delivers through asynchronous message queue?

The aggregate root's state is the product of the commands it receives. So different order of receiving the commands produce different aggregate root states. How can I check domain rules based on the ...
2
votes
0answers
128 views

Connect a web-app to to the back-end message queue

We are currently looking into dividing our back-end in multiple services using a message queue (RabbitMQ). Having a message queue suddenly gives us the possibility to update the web-app freely, but do ...
0
votes
1answer
234 views

How can an event subscriber catch up with missed events?

At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time. For example, a service is running and subscribes to events on ...
1
vote
1answer
242 views

Reduce duplicates in outbox pattern in event driven systems

Trying to implement the outbox pattern for an event driven system. The outbox pattern in a nutshell is a way to ensure system events are sent to the event log/queue/bus at least once (using the term ...
1
vote
2answers
108 views

Acceptable to use synchronous call to another microservice for time-sensitve state change?

Say there are two microservices (example is simplified) PickupRequestService: lists pick-up requests of passengers DriverService: drivers use to accept pickup requests On a completely decoupled ...
0
votes
1answer
73 views

Bringing data from 30 APIs based on data came from the APIs

I have to write server (.net core) that is going to read data from 30 different remote APIs. I suppose to run some sort of decision trees, if I find something in one API goes to another API, else some ...
3
votes
2answers
1k views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
1
vote
1answer
239 views

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
1
vote
1answer
82 views

Approach to handle network breakdowns in application dealing with database?

I have an application that writes data to database. In case of network breakdowns, the connection to database gets lost & my db queries freezes the whole application. So, in order to make it ...
0
votes
2answers
295 views

Should db commit and message-queue produce be one transaction?

I am making a microservice based architecture and I'm actively seeking antipatterns and trying to avoid them. In multiple scenarios, I have a database commit followed by a message that is produced to ...
0
votes
1answer
444 views

How to ensure persistence in an event-driven microservice architecture?

Right now we have a set of micro services with a client-facing gateway responsible for authentication and routing. MySQL database and RabbitMQ for the message queue. Internal services may queue up ...