Questions tagged [rabbitmq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
39 views

Integrating HTTP / Webhooks with Message Queues

I'm working at a project which integrates several Applications mostly SaaS Applications. The SaaS solutions have all the possibilities to hook into the internal event system with webhooks. The ...
0
votes
0answers
46 views

Is queue is the right choice for server request retries

I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. ...
1
vote
1answer
60 views

Best approach to update the User Interface on consumed event(s)?

I am developing a microservice architecture where the API publishes a message through RabbitMQ. One or more consumers will read the message, preform its functions then if successful acknowledge the ...
-3
votes
0answers
20 views

Scaling in P2P Kafka vs JMS/MQ

Consider using Kafka as a traditional point-to-point queue (by using just one consumer group). If you have, say 4 partitions of your Kafka topic, you can have a maximum of 4 consumers in your consumer ...
-2
votes
2answers
66 views

Inter-microservice communication with gRPC without hard-coding service endpoints

I'm building a microservices architecture that requires microservices to communicate with each other. Searching Google has led me to believe that gRPC is the preferred method for achieving this. ...
0
votes
0answers
16 views

Messaging - Responsibility of Consumer vs Publisher for Active-Active failover

Say there is an external client-DC1/DC2, posting requests to a service in DC1 and DC2 respectively. Now, if DC1 is down, should the external client-DC1 responsible for routing the request message to ...
1
vote
0answers
97 views

Approach to setup multi-tenant RabbitMQ workers in Python

I'm migrating my current application to multi-tenant setup. Now I've multiple rabbitmq workers to process async job, publish and consume integration events, and other stuffs. I'm planning to use ...
-2
votes
1answer
36 views

Synchronisation of different “channel” in an asynchronous way

I have a scenario where a service needs to update back and forth information from different channels (and each channel should have the same information, e.g inventory of products). It all works fine ...
0
votes
1answer
51 views

Process AMPQ messages both reliably and fast

I am new to RabbitMQ and herewith I want to make sure that I am not missing out on some advanced RabbitMQ feature or pattern I am not aware of. I need to develop a reliable system that processes a ...
0
votes
0answers
22 views

Difference in AMPS and AMQP

I recently across an acronym AMPS(Asyncronous Message Processing System( by 60EastTechnologies. Since it is not a standard like AMQP(Asynchronous Message Queuing Protocol) or JMS(Java Messaging System)...
0
votes
0answers
23 views

Broker Temporary Credentials for IoT Usage

I've several devices that need to send and receive data from a message broker, in my case RabbitMQ but is not so relevant for the discussion, and I need -as you can imagine- to provide connectivity in ...
0
votes
0answers
94 views

Why control the minimum and maximum concurrent consumers on an AMQP queue?

I'm trying here to understand why some frameworks introduce the control over the number of consumers on a AMQP queue. For example, Spring AMQP introduces this feature on the 1.3.0 version, with the ...
0
votes
2answers
119 views

Moving single tenant application with queue to multi tenants web application

I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
0
votes
3answers
917 views

Rabbitmq and multiple subscribers

Is it possible for a queue to have multiple consumers? I am working on a system where a single queue needs to be accessed by multiple consumers for different micro-services. Is it possible for the ...
0
votes
1answer
187 views

Implementing RESTful API in front of Event based microservices

I'm working on a system that implements multiple microservices which communicate via a RabbitMQ messaging bus. These microservices are created using python with the pika library (to publish messages ...
0
votes
0answers
30 views

RabbitMQ consumer for multiple services

I want to create single consumer application for multiple queues. I want to store queue names in config file, and stop listening to queues/exchanges that were removed from it(and start listening to ...
0
votes
2answers
304 views

How to send queue messages to one queue in rabbitmq?

I want to use rabbitmq to share domains events between microservices. Due to the fact that queues are round-robin, I need to define one queue per event source and event consumer. For example, cars-ms ...
1
vote
0answers
23 views

Consumer triggered entity data agregation

I am currently using RabbitMQ to aggregate the output of various data sources. Functionally the process is the following: The user enters a product name. The request is sent to the different sources ...
1
vote
1answer
58 views

How do I negotiate messages to avoid errors resulting from duplicate messages?

My system looks the following: I have N message producers, N RabbitMQ instances, N message consumers and 1 service. Producers produce the same messages while consumers consume the same messages. Here ...
2
votes
0answers
48 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
0
votes
1answer
273 views

Data replication with microservices using a messaging broker

I have a microservice based application like so: User.Microservice - stores , user information. Product.Microservice - stores products that user created. Order.Microservices - stores product orders ...
2
votes
0answers
146 views

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL ...
0
votes
1answer
302 views

What's the use of event sourcing and asynchronous message queues for sending commands

In short: Out of order commands end up with out of order events, so does the ordering of the events matter in such situations? Since we are uncertain about the order of commands when an asynchronous ...
5
votes
0answers
110 views

MQ integration: How to notify consumers of upcoming message format changes?

We have multiple microservices communicating over MQ. As MQ messages are the interface/contract between the services, whenever we make changes to the MQ message published by a service we need to make ...
0
votes
2answers
267 views

Best design for a AMQP work queue

I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will ...
1
vote
1answer
181 views

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find ...
1
vote
1answer
56 views

Queue based indexing & multiple languages

I'm thinking about refactoring my content indexing process to queues. Right know updates happen directly when editors change content. It is difficult to do maintance works on the search engine too ...
0
votes
0answers
39 views

AMQP integration

I’m trying to make a AMQP integration layer for my app. This layer would only consume messages from my customer’s queues and the result would be a specific action in my application. My application is ...
3
votes
0answers
133 views

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal ...
0
votes
1answer
603 views

RabbitMQ: how to achieve scalable one-to-one communication?

I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa: The servers stay always ...
2
votes
1answer
251 views

MQ Architecture: How to handle old messages in the queue and releasing a consumer/producer upgrade

So for example if I have a producer that sends a message {id: 1} to a consumer that expects that json object. And there's a ton of messages sitting in the queue waiting to be processed. I'm also ...
-1
votes
1answer
65 views

Writing a distributed chat logger

I'm trying to write a very high scale IRC logger that can handle like 1k messages per second. My Database of choice is cassandra and maybe later I might look into scylladb. My Question is how do I ...
2
votes
1answer
547 views

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they ...
6
votes
2answers
998 views

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
1
vote
3answers
1k views

Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?

My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
0
votes
1answer
504 views

asp.net Core 2 MVC - best way to send 1000s of emails every day

We need to send 1000s (under 10,000) per day for a few years. The data is in a SQL Server database - we have many asp.net MVC 5 and .net CORE 2 MVC sites on a Windows 2012 R2 server on IIS. The ...
1
vote
1answer
231 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...
3
votes
0answers
318 views

RabbitMQ: Topics and Persistence/Durability

Right now I'm using Direct Exchanges (i.e. one queue per "topic", pretty much pushing messages directly to a queue). This works OK, but I'd like the flexibility of working with Topic Exchanges, ...
1
vote
2answers
943 views

Architecture for Queuing based technology in a Micro-Service, Multi-Tenant Environment

Hopefully the title is not painfully obscure, but as it indicates, I'm really looking for advice on my architecture proposal, as the current architecture is subject to concurrency issues, performance ...
2
votes
1answer
200 views

What is the better place to create the Binding and Queue on RPC communication: Consumer or Producer?

I was talking with part of my team about the responsibility for who own the Binding and Queue creation. We are using Spring framework. In async communication they agree that the responsability to ...
8
votes
1answer
2k views

Making API calls with celery

I'm designing a system for a client where the requirements are: they upload a JSON file (one object/line) make a call to an API with the JSON object as the payload record the state (success/failure) ...