Questions tagged [rabbitmq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-2 votes
0 answers
27 views

Saga Choreography in Request - Reply MQ

Preface I'm Currently working on a project of the following architecture: API Gateway RabbitMQ Using the SAGA pattern. our language of choice is PHP. What we want to do: A route receives a request. ...
user avatar
  • 1
0 votes
1 answer
53 views

Long-running jobs in an event-driven environment with constrained max-execution-duration

Hello we have an async event-driven system (kotlin, spring cloud stream, rabbitmq) where there might be an event FooPayloadArrived, published by an ingress rest-controller. Processing this ...
user avatar
  • 115
-1 votes
2 answers
55 views

Applications interested to the same event and concurrency

I have an event pubblished in a bus(rabbit) and two application that are subscribed to this Event. public class CreationEvent { public int CreationEventID {get; set; } ... other properties } ...
user avatar
1 vote
1 answer
378 views

REST APIs vs Message brokers for Backend to Backend communication

We are a wholesale marketplace and we have three main products in our startup. The products are: 1- the marketplace app that allows customers to view products and purchase online. 2- the seller app ...
user avatar
0 votes
1 answer
99 views

What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?

I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic. How can ...
user avatar
0 votes
1 answer
168 views

Are RabbitMQ Streams overkill for 1 machine event-driven architecture aplication?

If developing event-driven application(more precisely event sourcing application) that will run solely on 1 machine are RabbitMQ Streams an overkill, or maybe a misfit? In theory RabbitMQ Streams seem ...
user avatar
2 votes
3 answers
624 views

RabbitMQ or REST API Public

Currently working on designing an application that will take in JSON data from remote client applications. The data these client applications are sending is going to go into a Message Broker, namely ...
user avatar
  • 21
0 votes
2 answers
64 views

Do I need a Message processing service before RabbitMQ (or any other message broker)

I hope that's a proper place to ask my question. I am wondering how is best to integrate RabbitMq (or any other message broker) into my project. There are 2 options: Simply collect all messages, ...
user avatar
  • 3
0 votes
1 answer
772 views

Rabbitmq create queues dynamically based on number of users

I have a scenario where I have to route a list of messages that I get to respective users. For suppose if I have messages = [ { text: 'hi', user_id: 1 }, { text: 'hi', user_id: 2 }, { text: 'hi',...
user avatar
0 votes
2 answers
418 views

How to share API between microservices?

Currently, our architecture uses an "API-first" approach in building our product. This product is divided across multiple teams handling different microservices. The above image succinctly ...
user avatar
0 votes
0 answers
222 views

High availability RabbitMQ using gRPC as transport layer?

I have an idea to use gRPC for all microservices communication for the sake of simplicity. Now there is a service that needs high availability, So the idea is to use RabitMQ as a message receiver from ...
user avatar
  • 647
2 votes
1 answer
234 views

How to decouple messaging middleware from microservices?

We have a bunch of microservices written in different programming languages. These all communicate with each other over RabbitMQ(AMQP) middleware. There are consumers and publishers in each ...
user avatar
0 votes
1 answer
87 views

Direct communication between message-driven bounded context

My project comprises of serveral bounded context which communicate with each other with RabbitMQ The front end of the project is written in C# and the backend is in Java. The RabbitMQ communication ...
user avatar
0 votes
0 answers
61 views

Redundancy approach for Pub-Sub application with events replication

I'm new to the microservices field. I'm looking for the right approaches regarding how to deal with the challenge of redundancy of a pus-sub application. Assume my application has has the below ...
user avatar
  • 101
0 votes
1 answer
35 views

How to dispatch sets of items to the relevant (LB) exclusive consumers with RabbitMQ?

I am trying to process sets of items (also, different sets have different items) in specific RabbitMQ consumers (one for each set of items) that would be created on-demand and are dispatched to the ...
user avatar
  • 311
-1 votes
1 answer
72 views

How to write package for two frameworks

I'm developing a package for Laravel & Lumen in the same code base, but I encountered some problems. Well, in the beginning, I thought it could have the same code and work for both frameworks, ...
user avatar
  • 17
1 vote
0 answers
192 views

RabbitMQ messages reliability for at-least-once delivery

I wanted to ask what are the possibilities of maintaining reliability when exchanging messages between microservices, when one of those message is rejected. As of today, we don't want to lose messages,...
user avatar
-1 votes
2 answers
569 views

Common library for classes and communication between microservices?

I have been working on a project that is being built around a microservices architecture. We are thinking on using a message broker, such as RabbitMQ, for both synchronous (via RPC) and asynchronous ...
user avatar
2 votes
1 answer
248 views

AMQP messaging: How to generate documentation for consumers?

My application publishes various messages (e.g., telemetry, changed-events, ...) to a RabbitMQ message-broker. The users have no access to the source-code and therefore, will need a separate ...
user avatar
0 votes
0 answers
206 views

Integrating HTTP / Webhooks with Message Queues

I'm working at a project which integrates several Applications mostly SaaS Applications. The SaaS solutions have all the possibilities to hook into the internal event system with webhooks. The ...
user avatar
  • 111
0 votes
0 answers
93 views

Is queue is the right choice for server request retries

I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. ...
user avatar
  • 109
1 vote
1 answer
67 views

Best approach to update the User Interface on consumed event(s)?

I am developing a microservice architecture where the API publishes a message through RabbitMQ. One or more consumers will read the message, preform its functions then if successful acknowledge the ...
user avatar
  • 121
-2 votes
2 answers
94 views

Inter-microservice communication with gRPC without hard-coding service endpoints

I'm building a microservices architecture that requires microservices to communicate with each other. Searching Google has led me to believe that gRPC is the preferred method for achieving this. ...
user avatar
  • 107
1 vote
0 answers
422 views

Approach to setup multi-tenant RabbitMQ workers in Python

I'm migrating my current application to multi-tenant setup. Now I've multiple rabbitmq workers to process async job, publish and consume integration events, and other stuffs. I'm planning to use ...
user avatar
-2 votes
1 answer
46 views

Synchronisation of different "channel" in an asynchronous way

I have a scenario where a service needs to update back and forth information from different channels (and each channel should have the same information, e.g inventory of products). It all works fine ...
user avatar
  • 107
0 votes
1 answer
66 views

Process AMPQ messages both reliably and fast

I am new to RabbitMQ and herewith I want to make sure that I am not missing out on some advanced RabbitMQ feature or pattern I am not aware of. I need to develop a reliable system that processes a ...
user avatar
0 votes
2 answers
381 views

Moving single tenant application with queue to multi tenants web application

I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
user avatar
0 votes
3 answers
8k views

Rabbitmq and multiple subscribers

Is it possible for a queue to have multiple consumers? I am working on a system where a single queue needs to be accessed by multiple consumers for different micro-services. Is it possible for the ...
user avatar
0 votes
2 answers
501 views

Implementing RESTful API in front of Event based microservices

I'm working on a system that implements multiple microservices which communicate via a RabbitMQ messaging bus. These microservices are created using python with the pika library (to publish messages ...
user avatar
0 votes
2 answers
871 views

How to send queue messages to one queue in rabbitmq?

I want to use rabbitmq to share domains events between microservices. Due to the fact that queues are round-robin, I need to define one queue per event source and event consumer. For example, cars-ms ...
user avatar
1 vote
0 answers
26 views

Consumer triggered entity data agregation

I am currently using RabbitMQ to aggregate the output of various data sources. Functionally the process is the following: The user enters a product name. The request is sent to the different sources ...
user avatar
  • 119
1 vote
1 answer
109 views

How do I negotiate messages to avoid errors resulting from duplicate messages?

My system looks the following: I have N message producers, N RabbitMQ instances, N message consumers and 1 service. Producers produce the same messages while consumers consume the same messages. Here ...
user avatar
  • 111
2 votes
0 answers
69 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
user avatar
0 votes
1 answer
612 views

Data replication with microservices using a messaging broker

I have a microservice based application like so: User.Microservice - stores , user information. Product.Microservice - stores products that user created. Order.Microservices - stores product orders ...
user avatar
2 votes
0 answers
273 views

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL ...
user avatar
  • 151
0 votes
1 answer
644 views

What's the use of event sourcing and asynchronous message queues for sending commands

In short: Out of order commands end up with out of order events, so does the ordering of the events matter in such situations? Since we are uncertain about the order of commands when an asynchronous ...
user avatar
  • 209
5 votes
1 answer
164 views

MQ integration: How to notify consumers of upcoming message format changes?

We have multiple microservices communicating over MQ. As MQ messages are the interface/contract between the services, whenever we make changes to the MQ message published by a service we need to make ...
user avatar
  • 151
0 votes
2 answers
958 views

Best design for a AMQP work queue

I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will ...
user avatar
  • 1
1 vote
2 answers
499 views

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
user avatar
1 vote
1 answer
86 views

Queue based indexing & multiple languages

I'm thinking about refactoring my content indexing process to queues. Right know updates happen directly when editors change content. It is difficult to do maintance works on the search engine too ...
user avatar
  • 197
0 votes
0 answers
40 views

AMQP integration

I’m trying to make a AMQP integration layer for my app. This layer would only consume messages from my customer’s queues and the result would be a specific action in my application. My application is ...
user avatar
3 votes
1 answer
207 views

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal ...
user avatar
  • 139
1 vote
1 answer
978 views

RabbitMQ: how to achieve scalable one-to-one communication?

I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa: The servers stay always ...
user avatar
  • 161
2 votes
1 answer
336 views

MQ Architecture: How to handle old messages in the queue and releasing a consumer/producer upgrade

So for example if I have a producer that sends a message {id: 1} to a consumer that expects that json object. And there's a ton of messages sitting in the queue waiting to be processed. I'm also ...
user avatar
  • 123
-1 votes
1 answer
68 views

Writing a distributed chat logger

I'm trying to write a very high scale IRC logger that can handle like 1k messages per second. My Database of choice is cassandra and maybe later I might look into scylladb. My Question is how do I ...
user avatar
  • 101
2 votes
1 answer
853 views

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they ...
user avatar
  • 397
7 votes
2 answers
2k views

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
user avatar
2 votes
3 answers
2k views

Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?

My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
user avatar
  • 2,257
0 votes
1 answer
565 views

asp.net Core 2 MVC - best way to send 1000s of emails every day

We need to send 1000s (under 10,000) per day for a few years. The data is in a SQL Server database - we have many asp.net MVC 5 and .net CORE 2 MVC sites on a Windows 2012 R2 server on IIS. The ...
user avatar
  • 241
1 vote
1 answer
249 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...
user avatar