Questions tagged [message-queue]

A message queue is a service that receives events or messages from producers and provides them to consumers.

Filter by
Sorted by
Tagged with
7 votes
2 answers
2k views

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
user avatar
2 votes
3 answers
2k views

Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?

My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
user avatar
  • 2,267
1 vote
1 answer
262 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...
user avatar
2 votes
1 answer
451 views

RabbitMQ: Topics and Persistence/Durability

Right now I'm using Direct Exchanges (i.e. one queue per "topic", pretty much pushing messages directly to a queue). This works OK, but I'd like the flexibility of working with Topic Exchanges, ...
user avatar
1 vote
0 answers
97 views

Optimistic locking vs separate lock store for background workers

I need to compute the value of a row R on a table based on rows queried on another table B. I need to do it in a queue worker because the computation of the value can be slow. A message is queued ...
user avatar
1 vote
2 answers
1k views

Architecture for Queuing based technology in a Micro-Service, Multi-Tenant Environment

Hopefully the title is not painfully obscure, but as it indicates, I'm really looking for advice on my architecture proposal, as the current architecture is subject to concurrency issues, performance ...
user avatar
2 votes
1 answer
222 views

What is the better place to create the Binding and Queue on RPC communication: Consumer or Producer?

I was talking with part of my team about the responsibility for who own the Binding and Queue creation. We are using Spring framework. In async communication they agree that the responsability to ...
user avatar
  • 2,267
0 votes
1 answer
137 views

Where should I store statistics about sent emails?

my application sends a batch of custom emails every day using mailgun (an email API service). I have a producer worker scheduled to run daily, and all does is query the database for the unique ...
user avatar
8 votes
5 answers
1k views

Message validation in async messaging-based services

I'm looking for information on the best approach to message validation in asynchronous-messaging-based services (i.e. services that pull messages from some sort of message queue or broker, rather than ...
user avatar
  • 1,045
-2 votes
1 answer
508 views

PHP or NodeJS for a chat app with message queue

In my company I proposed NodeJS for developing a microservice that acts as a bridge between the front-end and a third party chat API, that Microservice will receive large amounts of messages (about ...
user avatar
  • 127
0 votes
1 answer
665 views

Queue - move message from one queue to another after expiration

I have producer / worker queue (1 producer, many workers). Currently using Redis, but I dont mind switching to RabbitMQ or anything else. Worker takes a task from queue, runs long running job and ...
user avatar
  • 111
1 vote
1 answer
270 views

How do I set up short-lived queues?

This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-...
user avatar
  • 1,080
1 vote
1 answer
130 views

Microservice Communication: Enforce RPC?

I have the opinion that each method in a service should only do one small step of a larger task, delegate a result to the successive step/method and terminate. No matter if this next method lives in ...
user avatar
  • 11
3 votes
0 answers
724 views

Designing CRUD Messaging communication

There are endless resources on creating CRUD for REST resources but I can't find much on doing the same for Messaging. Given two services A and B where A receives incoming requests that initiates the ...
user avatar
  • 415
5 votes
1 answer
1k views

How to deal with failing messages in DDD?

We have a set of micro-services all built according to the Domain Driven Design (DDD). The micro-services communicate via Domain Events with each other (OrderSubmittedEvent, CustomerBilledEvent, ...). ...
user avatar
0 votes
1 answer
49 views

Queued work serialized by entity, notifications to UI

We're having a lot of fun and success using Azure Functions, but looking for advice on two related design issues. Our first challenge is that we need to communicate job state back to the UI. So the ...
user avatar
  • 145
2 votes
2 answers
4k views

Replacing database polling with messaging queues

We have a background job(java batch) which listens/reads from an MQ and persists a message onto the database(local table). Then a trigger sends this message into a queuing table in WAITING status. My ...
user avatar
  • 81
3 votes
1 answer
966 views

Should we be using pub/sub in our messaging stack?

Basically this question is of the variety "Can I check that we're not about to do something dumb?" We're setting up a new system that needs to guarantee delivery of emails. We have an ...
user avatar
  • 269
0 votes
2 answers
1k views

How can I create a Python microservice on AWS that both accepts REST connections and processes SQS messages?

Background: I'm working on a new project at work that will run in AWS. We're trying to use a modern microservice architecture and take advantage of cloud technology, but we don't have much experience ...
user avatar
  • 3
3 votes
1 answer
210 views

How to organize exchanges of non critical messages between servers in multiple data centers by using Kafka (or other solution)?

We need to organize a way to exchange messages between servers in multiple data centers. Messages are not critical. We just need to be able to send messages from any servers in any data centers to any ...
user avatar
  • 141
1 vote
0 answers
353 views

concurrency via message queues: how to avoid recursion?

I am working on a concurrency system based on message queues with an underlying thread pool. A queue can target other queues. Also a queue can be targeted by multiple queues (i.e. children) to create ...
user avatar
0 votes
4 answers
205 views

How to implement "viewed: N times" functionality for an article?

How will I have to update "views_count" each time(!) a user loads my page? It will be costly. And I don't need such accuracy when I update "views_count" after a user loads a page. Then is there any ...
user avatar
  • 67
0 votes
1 answer
722 views

Design Pattern similar to ESB

Background I am trying to write a simulator where multiple AI agents are competing and/or collaborating to achieve the goal of maximizing some utility function. Each agent has the ability to interact ...
user avatar
  • 584
0 votes
1 answer
166 views

How to serve a computationally intensive application via REST? [closed]

I have got a native image processing application that can enhance photos(peApp). Now, I want to serve this functionality to the world by having a server application that supports REST. Now while ...
user avatar
1 vote
1 answer
564 views

TPL parallel for each or custom queue for multithreaded data processing

I'm looking to do the following for 1000s of items: 1) get time series data for an item from file 2) calculate mean and standard deviation 3) calculate final calculation using mean and standard ...
user avatar
1 vote
1 answer
324 views

Is my approach at building middleware using a queue a good approach?

I want to build a middleware system that my company can use and expand on to handle the integration points between our various bespoke and 3rd party systems as well as our hosted websites. I am ...
user avatar
  • 349
4 votes
1 answer
3k views

Architecture for real-time application, avoiding duplicate messages for queues with no active consumers

I am designing the architecture of an application which has your typical backend and frontend components. The frontend application must be semi real-time. For this requirement, the backend is pushing ...
user avatar
3 votes
1 answer
2k views

RabbitMQ Race conditions / Dependent messages

I'm currently working on a project, where we'll have to keep a local system in "sync" with a remote application. For example if a new customer is created in the local application, this customer must ...
user avatar
27 votes
5 answers
17k views

Message Queue. Database vs Dedicated MQ

I am after advice regarding message queueing. We have requirements for "jobs" to be posted to a message queue. The original suggestion was just to use a SQL Server instance and process messages from ...
user avatar
4 votes
1 answer
834 views

distributed application design - using message broker

Background I'm trying to design a layer/component in my distributed application that will communicate between component A and component B. Right now, this "communication" is accomplished by ...
user avatar
  • 479
1 vote
1 answer
1k views

What is the advantage of POSIX queue or just normal data structure queue?

I'm programming with the embedded device running Linux, where memory is less, I just have 64MB flash only. I planned to use queues for the thread communication. In which I came across with using ...
user avatar
17 votes
2 answers
7k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
user avatar
  • 4,680
6 votes
3 answers
7k views

Implementing a caching microservice by avoiding potential bottlenecks

The system I am working on employs quite a sophisticated caching/preloading mechanism for external third-parties, and since it is built on a microservice architecture, I would like to extract the ...
user avatar
  • 161
2 votes
2 answers
3k views

Solve Synchronization

I want to solve Synchronization with a Message Queue In the wikipedia page of message queue, I read: Most messaging systems support both the publisher/subscriber and message queue models in their ...
user avatar
  • 180
5 votes
2 answers
138 views

How to structure messages between a web server and a job server

For the SaaS infrastructure I'm building, we need to send transactional emails and run other long-running tasks on a separate job server. I can communicate via a queue, such as AWS' SQS or via a ...
user avatar
0 votes
1 answer
151 views

Message queues and triggered comms within a native app

I'm building a native app which currently has zero backend infrastructure. With services like Firebase's Authentication, database and notifications means that all interactions are handled client-side ...
user avatar
  • 31
4 votes
1 answer
1k views

Message driven architecture and horizontal scaling

During working on a pet project in microservice event driven architecture, I've faced a problem of delivering responses to a client. Each microservice (API Gateway etc) has many replicas and has it's ...
user avatar
  • 431
8 votes
2 answers
729 views

Microservices - compensate service failure with queue

We are using some sort of microservices approach in our app (although it is not really consequently adhered to). When a service is down or throwing an exception, the approach is to put it into a ...
user avatar
4 votes
1 answer
115 views

Message publishing interface when deferring until transaction is committed

Suppose I have the following interface used for publishing messages onto a message queue: interface IMessageProducer { void Publish<TMessage>(TMessage message); } Normally, an implementer ...
user avatar
4 votes
1 answer
3k views

AMQP routing-keys naming anti-patterns when using topics

I'm working on the specification for building a web services platform and AMQP is the messaging protocol used for the interaction between the components of this platform. I'm using topic exchanges ...
user avatar
  • 151
0 votes
0 answers
50 views

Synchronized computation task distribution

I have a system where a set of Operations needs to be done every day. Each Operation has the following metadata: Type, Entity and a Date (independent of current date). It reads something from the ...
user avatar
0 votes
1 answer
125 views

Where should I store stream wide metadata in stream processing?

I am building a stream processing architecture and was wondering what to do with metadata about the stream. For instance, every message of data coming from a source has the same attribution, as it ...
user avatar
3 votes
1 answer
401 views

How to recognize consumers in AMQP architecture

I'm doing an project where I'm using a microservice architecture. All the services within this architecture communicate with each other using an AMQP broker (RabbitMQ). When something happens in a ...
user avatar
  • 191
3 votes
1 answer
1k views

Using messaging queue as async mysql writes

I read some articles about developers who takes rabbitmq and produce from PHP messages to write to mysql. they do this to make the page speedup and not wait for mysql to return an answer. So in my ...
user avatar
  • 185
9 votes
4 answers
2k views

Handling changes in a event-driven microservice architecture

I'm doing an research-project where I'm researching the options to handle changes in an event-driven microservice architecture. So, let's say we got an application where we got four different ...
user avatar
  • 191
3 votes
1 answer
187 views

Best practice for delaying jobs

I have a system in PHP that works like a booking system. Once a booking is confirmed there are a total of 7 emails/sms that needs to be send. The times of when they need to be send are already known. ...
user avatar
  • 171
0 votes
2 answers
1k views

How can an update message from a rabbitmq broker reach all relevant clients via multiple websocket servers?

Lets say I have multiple websocket servers that maintain many connections to clients to send updates. These websocket servers will be pulling said updates from a rabbitmq broker, and broadcasting them ...
user avatar
  • 103
2 votes
1 answer
151 views

What pattern should I set for handling event as working item?

I have this business event : "CustomerUpdated" published by my system. I have many application listening to these events, and they do all king of action : sending email, updating the database, ...
user avatar
6 votes
3 answers
946 views

Slow throughput - still worth using a dedicated message queue?

Here are some requirements for a queue: Every few days add ~100k tasks with various priorities Workers will pull tasks at typically less than 10 / second Tasks need to be completed by ~2 unique ...
user avatar
  • 163
6 votes
2 answers
727 views

Handling cases where the message queue is unreachable

We are using Apache Camel with ActiveMQ in a microservice architecture in a growing system of integration solutions. I was wondering how each service should react when they are unable to connect to ...
user avatar
  • 135