Questions tagged [message-queue]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

What is the recommended way to start a long-running containerized analysis in R from NodeJS?

I have a frontend through which users can submit analysis requests, which are inserted into a database. These analysis requests are received in a NodeJS server using Postgres notify/listen. Using ...
0
votes
0answers
39 views

Integrating HTTP / Webhooks with Message Queues

I'm working at a project which integrates several Applications mostly SaaS Applications. The SaaS solutions have all the possibilities to hook into the internal event system with webhooks. The ...
1
vote
0answers
23 views

How to process a long list of data from database with Message-driven bean due to the asynchronous requirement properly

There is a task in our JavaEE project: a user in the frontend page A can click a button to request to get a zip with the processed data. As the process of the request in the backend needs plenty of ...
1
vote
1answer
75 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
0
votes
0answers
46 views

Is queue is the right choice for server request retries

I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. ...
-3
votes
0answers
20 views

Scaling in P2P Kafka vs JMS/MQ

Consider using Kafka as a traditional point-to-point queue (by using just one consumer group). If you have, say 4 partitions of your Kafka topic, you can have a maximum of 4 consumers in your consumer ...
0
votes
0answers
8 views

Scale spring-batch instances executed from within a web container

I am re-designing our current ETL for file ingestion where spring-batch containers will be deployed on Kubernetes. Each file type is processed by a different job, which will be triggered on demand via ...
0
votes
0answers
15 views

Kafka with log compaction best approach

I am publishing entities with entity-id (message-key) + version in kafka. I decided to use log compaction to ensure that atleast 1 version of the entity is available at all times and for that to be ...
0
votes
0answers
16 views

Messaging - Responsibility of Consumer vs Publisher for Active-Active failover

Say there is an external client-DC1/DC2, posting requests to a service in DC1 and DC2 respectively. Now, if DC1 is down, should the external client-DC1 responsible for routing the request message to ...
0
votes
1answer
54 views

Question about the best way to architect a small webapp

So I'm trying to create a simple website where I run a computationally intensive task on the backend based on input submitted by a form. For reference: my goal is not too dissimilar from something ...
0
votes
0answers
35 views

Using a message queue or pub / sub solution for connecting frontend to backend instead of REST APIs

I own a dell r620 server at home and want to make more use of it. Typically its my staging environment as my internet is currently ipv6 with dslite so I can’t port forward. But I can easily call and ...
1
vote
1answer
31 views

optimal way of dealing with subscriptions

I have a system in which some data is inserted into database. Whenever that data is inserted subscribers should be notified about that (it doesn't have to happen just after inserting, but there is a ...
1
vote
1answer
32 views

Designing Kafka topics for secured event driven job scheduling system

Consider the following Group A Job A { Depends on Job B of Group A Run User -> User1 } Job B { Depends on Job C and Job D of Group A Run User -> User2 } Job C { Depends ...
0
votes
0answers
17 views

Porting a SYSV IPC multiprocess architecture to Windows

I'm looking at porting a Unix-based multi-process architecture server application. The app uses SYS V IPC shared memory and semaphores. It has a main message queue in shared memory, with semaphores ...
0
votes
0answers
19 views

Can a common ActiveMQ service be shared in a dev environment without messages being consumed between developers?

Our dev environment is currently sharing an ActiveMQ service hosted on AWS. This however resulted in problems where dev A's service produce a message that will trigger dev B's localhost consumer. Is ...
0
votes
1answer
51 views

Process AMPQ messages both reliably and fast

I am new to RabbitMQ and herewith I want to make sure that I am not missing out on some advanced RabbitMQ feature or pattern I am not aware of. I need to develop a reliable system that processes a ...
0
votes
0answers
22 views

Difference in AMPS and AMQP

I recently across an acronym AMPS(Asyncronous Message Processing System( by 60EastTechnologies. Since it is not a standard like AMQP(Asynchronous Message Queuing Protocol) or JMS(Java Messaging System)...
0
votes
0answers
92 views

Why control the minimum and maximum concurrent consumers on an AMQP queue?

I'm trying here to understand why some frameworks introduce the control over the number of consumers on a AMQP queue. For example, Spring AMQP introduces this feature on the 1.3.0 version, with the ...
0
votes
1answer
133 views

Integrating with real-time data from multiple devices with accelerometer

I am designing a MVP for a simple gamification system for a trampoline park(s). External company is providing bracelets for customers, then collecting this data in their own web app, where they ...
3
votes
2answers
120 views

Communicating bulk data among microservices

Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication. My use case is to import data from upstream feed files(say 50k records) to our ...
0
votes
3answers
92 views

Keeping up logical order in a messaging environment

My setting is the following (Please assume that this points are entirely rock solid unchangeable, some for good reasons some: "just because"): In a super scalable microservice environment i receive ...
0
votes
2answers
119 views

Moving single tenant application with queue to multi tenants web application

I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
0
votes
0answers
12 views

Distribute processing load across multiple nodes of same Web Application

We have an application deployed across multiple nodes. There is a scheduled job inside this application that would be run on any one of the nodes. When the job is triggered it is supposed to read ...
1
vote
1answer
68 views

Alternative architecture for multi-instance, client-to-client message relay/broker

Research leads me to believe the most common architecture for a multi-instance (horizontally-scaling) message relay service (the typical example being an instant messenger/chat service) is to ...
3
votes
1answer
806 views

Do I actually need a message broker or are websockets enough?

The website I am building has a real-time messaging component. The backend is built with Flask and I have integrated Flask-SocketIO to handle Websocket connections when users are on the messaging page....
0
votes
2answers
92 views

Possible microservice based design

Here are the design considerations for a software. The querying of data should be as fast as possible (reader <1s, writer <2.5s possibly since it also take time to reach the client end web or ...
0
votes
0answers
12 views

Technique to match a message by criteria to node in the cluster

I have global message queue with multiple different producers, and a cluster with N nodes of consumers. Each consumer receives a message and may decide to process it or to put it back to the queue if ...
0
votes
2answers
323 views

What is the Difference between Reactive Extensions and Message Queue?

I am trying to understand the difference between Reactive Extensions and Message Queue? Are they competing frameworks? Can they be used in conjuction? Started programming, and trying to understand ...
1
vote
1answer
75 views

Best practice for multi-application communication and events?

Essentially, I have a web application and a console application. The web application allows users to configure what will be processed, which is just a big information processor. From the web UI, a ...
1
vote
2answers
333 views

Message Queue with multiple consumers locking synchronous on a field

We have a system where multiple consumers get messages out of a queue (currently implemented with a database). The messages have an order field. All messages of one particular order should be ...
0
votes
0answers
75 views

Are message identifiers and offsets the same thing in Kafka?

I am reading up on Kafka and encountered the following on one of the tutorials. The producer sends the following to the broker - unique message identifier, timestamp and a binary payload. The ...
2
votes
0answers
144 views

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL ...
3
votes
1answer
151 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0
votes
2answers
76 views

How to check if the command is execuable when it delivers through asynchronous message queue?

The aggregate root's state is the product of the commands it receives. So different order of receiving the commands produce different aggregate root states. How can I check domain rules based on the ...
2
votes
0answers
102 views

Connect a web-app to to the back-end message queue

We are currently looking into dividing our back-end in multiple services using a message queue (RabbitMQ). Having a message queue suddenly gives us the possibility to update the web-app freely, but do ...
0
votes
1answer
170 views

How can an event subscriber catch up with missed events?

At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time. For example, a service is running and subscribes to events on ...
1
vote
1answer
200 views

Reduce duplicates in outbox pattern in event driven systems

Trying to implement the outbox pattern for an event driven system. The outbox pattern in a nutshell is a way to ensure system events are sent to the event log/queue/bus at least once (using the term ...
1
vote
2answers
87 views

Acceptable to use synchronous call to another microservice for time-sensitve state change?

Say there are two microservices (example is simplified) PickupRequestService: lists pick-up requests of passengers DriverService: drivers use to accept pickup requests On a completely decoupled ...
0
votes
1answer
71 views

Bringing data from 30 APIs based on data came from the APIs

I have to write server (.net core) that is going to read data from 30 different remote APIs. I suppose to run some sort of decision trees, if I find something in one API goes to another API, else some ...
1
vote
1answer
674 views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
1
vote
1answer
180 views

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
1
vote
1answer
82 views

Approach to handle network breakdowns in application dealing with database?

I have an application that writes data to database. In case of network breakdowns, the connection to database gets lost & my db queries freezes the whole application. So, in order to make it ...
0
votes
2answers
215 views

Should db commit and message-queue produce be one transaction?

I am making a microservice based architecture and I'm actively seeking antipatterns and trying to avoid them. In multiple scenarios, I have a database commit followed by a message that is produced to ...
0
votes
1answer
318 views

How to ensure persistence in an event-driven microservice architecture?

Right now we have a set of micro services with a client-facing gateway responsible for authentication and routing. MySQL database and RabbitMQ for the message queue. Internal services may queue up ...
5
votes
2answers
1k views

Synchronization of data across microservices

We have 2 to 3 dozen microservices that serve our customers. These services are deployed in a Kubernetes cluster, and they're only accessible to the outside world through 3 or 4 API gateways. We ...
0
votes
1answer
79 views

How to implement adaptive routing?

I need to send jobs to worker queues according to resource usage and other metrics like how many jobs have been processed and how successful they were. There is also the need to “weight” these ...
0
votes
1answer
176 views

Which message pattern should I use?

I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
0
votes
0answers
325 views

Using signalR for streaming alongside a microservice architecture

I have a cluster of micro services with an exposed search API serving a client app. I want to "stream" in real time the aggregated results of a complex search. The search will be kicked of by the ...
3
votes
3answers
250 views

Dealing with data arriving at a different times

I have a load balanced service that needs two data inputs to correctly give a result. One of these inputs comes from a queue, the other comes from user input. Most of the time the data that comes ...
0
votes
1answer
297 views

Scheduling operations in a micro service

I'm trying to realize a service that executes operations at a certain time, those operations are specified by another service by messaging, here's the scenario: The service A receives a request to ...