Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [message-queue]

The tag has no usage guidance.

0
votes
1answer
25 views

Queuing emails in Laravel when using an api driver such as Mailgun?

In Laravel (for the record I'm using v5.6) there is an option to queue emails for background processing. I have a scheduled job to send out daily email alerts to users. Currently the job drops the ...
0
votes
0answers
11 views

How to handle S3 style bucket notifications with ceph

The ceph s3 api doesn't support notifications like S3 or minio do. How would you implement notifications for ceph object storage based on new uploads / deletes? Is rolling my own cron job basically ...
2
votes
1answer
40 views

MQ Architecture: How to handle old messages in the queue and releasing a consumer/producer upgrade

So for example if I have a producer that sends a message {id: 1} to a consumer that expects that json object. And there's a ton of messages sitting in the queue waiting to be processed. I'm also ...
2
votes
1answer
41 views

Decomposition by “role” or by “usage” - is that a thing?

I have a mail_queue table that is written to by web requests then worked on by a worker to send the mails. I've now had a moment of enlightenment re. what I was not comfortable with in the design. ...
0
votes
1answer
150 views

Message Bus v Mediator pattern v In Memory Bus

I am struggling to understand when to use each of these: 1) Message Bus: Used to send integration events between Microservices. For example, Microservice A could publish an integration event, which ...
0
votes
1answer
59 views

How an async response is usually getting back to the clent in typical reactive highload architecture?

A typical highload reactive architecture is: (Q means queue) And usually arrows are from client to DB at such sketches. My question is how the response is going back? The same queues and other stuff?...
0
votes
1answer
122 views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
3
votes
1answer
170 views

How to handle concurrent updates for the same record in a distributed system

I'm trying to figure out the best solution for the below. Any help would be great. So basically I have a service (that can be scaled horizontally), which listens on a queue. Every message received ...
1
vote
0answers
178 views

Message Queues and Multiple Subscribers

My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at ...
-3
votes
1answer
31 views

Disable recieving messages from queue remotly

I have a CRUD like service which have REST HTTP API. This service have two instances - only one recieving request at time. If first instance starts throwing exceptions or slowing down we change load ...
2
votes
2answers
150 views

When consuming an api, what is a good way to deal with their request limits?

My app is consuming a third party api. One of the requirements of this api is that my app cannot send more than 20 requests per second. Because of the nature of this app, and because my user base is ...
0
votes
1answer
171 views

How to load test Publisher/Subscribing queue?

I am trying to load test a queue that works on a pub sub pattern and can't figure out how response time is to be calculate and the TPS(transactions per second). Problem statement : The pub sub ...
0
votes
1answer
114 views

Architecture pattern for temporary message queue subscriptions with websocket clients

We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
5
votes
2answers
311 views

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
1
vote
2answers
239 views

Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?

My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
1
vote
1answer
46 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...
2
votes
0answers
187 views

RabbitMQ: Topics and Persistence/Durability

Right now I'm using Direct Exchanges (i.e. one queue per "topic", pretty much pushing messages directly to a queue). This works OK, but I'd like the flexibility of working with Topic Exchanges, ...
1
vote
0answers
59 views

Optimistic locking vs separate lock store for background workers

I need to compute the value of a row R on a table based on rows queried on another table B. I need to do it in a queue worker because the computation of the value can be slow. A message is queued ...
1
vote
2answers
453 views

Architecture for Queuing based technology in a Micro-Service, Multi-Tenant Environment

Hopefully the title is not painfully obscure, but as it indicates, I'm really looking for advice on my architecture proposal, as the current architecture is subject to concurrency issues, performance ...
2
votes
1answer
174 views

What is the better place to create the Binding and Queue on RPC communication: Consumer or Producer?

I was talking with part of my team about the responsibility for who own the Binding and Queue creation. We are using Spring framework. In async communication they agree that the responsability to ...
0
votes
1answer
54 views

Where should I store statistics about sent emails?

my application sends a batch of custom emails every day using mailgun (an email API service). I have a producer worker scheduled to run daily, and all does is query the database for the unique ...
6
votes
5answers
303 views

Message validation in async messaging-based services

I'm looking for information on the best approach to message validation in asynchronous-messaging-based services (i.e. services that pull messages from some sort of message queue or broker, rather than ...
-2
votes
1answer
291 views

PHP or NodeJS for a chat app with message queue

In my company I proposed NodeJS for developing a microservice that acts as a bridge between the front-end and a third party chat API, that Microservice will receive large amounts of messages (about ...
0
votes
1answer
209 views

Queue - move message from one queue to another after expiration

I have producer / worker queue (1 producer, many workers). Currently using Redis, but I dont mind switching to RabbitMQ or anything else. Worker takes a task from queue, runs long running job and ...
1
vote
1answer
120 views

How do I set up short-lived queues?

This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-...
1
vote
1answer
95 views

Microservice Communication: Enforce RPC?

I have the opinion that each method in a service should only do one small step of a larger task, delegate a result to the successive step/method and terminate. No matter if this next method lives in ...
1
vote
0answers
178 views

Designing CRUD Messaging communication

There are endless resources on creating CRUD for REST resources but I can't find much on doing the same for Messaging. Given two services A and B where A receives incoming requests that initiates the ...
5
votes
1answer
518 views

How to deal with failing messages in DDD?

We have a set of micro-services all built according to the Domain Driven Design (DDD). The micro-services communicate via Domain Events with each other (OrderSubmittedEvent, CustomerBilledEvent, ...). ...
1
vote
1answer
33 views

Queued work serialized by entity, notifications to UI

We're having a lot of fun and success using Azure Functions, but looking for advice on two related design issues. Our first challenge is that we need to communicate job state back to the UI. So the ...
3
votes
2answers
1k views

Replacing database polling with messaging queues

We have a background job(java batch) which listens/reads from an MQ and persists a message onto the database(local table). Then a trigger sends this message into a queuing table in WAITING status. My ...
1
vote
1answer
221 views

Should we be using pub/sub in our messaging stack?

Basically this question is of the variety "Can I check that we're not about to do something dumb?" We're setting up a new system that needs to guarantee delivery of emails. We have an existing ...
0
votes
2answers
482 views

How can I create a Python microservice on AWS that both accepts REST connections and processes SQS messages?

Background: I'm working on a new project at work that will run in AWS. We're trying to use a modern microservice architecture and take advantage of cloud technology, but we don't have much experience ...
1
vote
0answers
58 views

Designing a configurable delay pattern on JMS (tibco maas) stack

My java app is a consumer to maas topic(say PRIMARY) on which upstream systems publishes their messages. Some of the pushed messages now needs some delay(say 10 min) before my app starts processing it....
3
votes
1answer
112 views

How to organize exchanges of non critical messages between servers in multiple data centers by using Kafka (or other solution)?

We need to organize a way to exchange messages between servers in multiple data centers. Messages are not critical. We just need to be able to send messages from any servers in any data centers to any ...
1
vote
0answers
110 views

concurrency via message queues: how to avoid recursion?

I am working on a concurrency system based on message queues with an underlying thread pool. A queue can target other queues. Also a queue can be targeted by multiple queues (i.e. children) to create ...
0
votes
4answers
196 views

How to implement “viewed: N times” functionality for an article?

How will I have to update "views_count" each time(!) a user loads my page? It will be costly. And I don't need such accuracy when I update "views_count" after a user loads a page. Then is there any ...
0
votes
1answer
381 views

Design Pattern similar to ESB

Background I am trying to write a simulator where multiple AI agents are competing and/or collaborating to achieve the goal of maximizing some utility function. Each agent has the ability to ...
0
votes
1answer
123 views

How to serve a computationally intensive application via REST? [closed]

I have got a native image processing application that can enhance photos(peApp). Now, I want to serve this functionality to the world by having a server application that supports REST. Now while ...
1
vote
1answer
439 views

TPL parallel for each or custom queue for multithreaded data processing

I'm looking to do the following for 1000s of items: 1) get time series data for an item from file 2) calculate mean and standard deviation 3) calculate final calculation using mean and standard ...
1
vote
1answer
144 views

Is my approach at building middleware using a queue a good approach?

I want to build a middleware system that my company can use and expand on to handle the integration points between our various bespoke and 3rd party systems as well as our hosted websites. I am ...
3
votes
1answer
1k views

Architecture for real-time application, avoiding duplicate messages for queues with no active consumers

I am designing the architecture of an application which has your typical backend and frontend components. The frontend application must be semi real-time. For this requirement, the backend is pushing ...
1
vote
1answer
716 views

RabbitMQ Race conditions / Dependent messages

I'm currently working on a project, where we'll have to keep a local system in "sync" with a remote application. For example if a new customer is created in the local application, this customer must ...
10
votes
5answers
3k views

Message Queue. Database vs Dedicated MQ

I am after advice regarding message queueing. We have requirements for "jobs" to be posted to a message queue. The original suggestion was just to use a SQL Server instance and process messages from ...
4
votes
1answer
469 views

distributed application design - using message broker

Background I'm trying to design a layer/component in my distributed application that will communicate between component A and component B. Right now, this "communication" is accomplished by ...
1
vote
1answer
505 views

What know the advantage of POSIX queue or just normal data structure queue?

I'm programming with the embedded device running Linux, where memory is less, I just have 64MB flash only. I planned to use queues for the thread communication. In which I came across with using ...
7
votes
2answers
4k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
1
vote
0answers
35 views

Can a message broker be used with a hook, extension point, or plugin design?

When designing a plugin system with publishers and subscribers, is it possible to have a message broker be the manager of the "extension point"? For example, if I wrote a program and it had a hook ...
3
votes
3answers
4k views

Implementing a caching microservice by avoiding potential bottlenecks

The system I am working on employs quite a sophisticated caching/preloading mechanism for external third-parties, and since it is built on a microservice architecture, I would like to extract the ...
2
votes
2answers
1k views

Solve Synchronization

I want to solve Synchronization with a Message Queue In the wikipedia page of message queue, I read: Most messaging systems support both the publisher/subscriber and message queue models in their ...
4
votes
2answers
111 views

How to structure messages between a web server and a job server

For the SaaS infrastructure I'm building, we need to send transactional emails and run other long-running tasks on a separate job server. I can communicate via a queue, such as AWS' SQS or via a ...