Questions tagged [message-queue]

A message queue is a service that receives events or messages from producers and provides them to consumers.

Filter by
Sorted by
Tagged with
6 votes
1 answer
767 views

Best practice for ensuring name uniqueness/correctness in message queues in a microservice architecture/distributed system

I was wondering what a good solution would be for ensuring that queue-names are entered correctly and are only used by the correct applications in a large system which uses message queues to exchange ...
user avatar
  • 135
0 votes
1 answer
542 views

A WebSocket API, reply message

I am writing a Web application for copying files between servers. I decided to write a WebSocket server which will receive and send relevant messages (that is communicate with a browser), start and ...
user avatar
  • 722
2 votes
2 answers
105 views

AMQP broker and happen-before guarantees

I am trying to figure out how an AMQP broker, such as RabbitMQ, can be integrated in our architecture. This should allow to easily scale horizontally in the future. For the sake of simplicity, let me ...
user avatar
3 votes
1 answer
2k views

Web API to Windows Service communication via ZeroMQ

I have an ASP.NET Web API 2 web service. This is the interface customers will use to submit data. I have a Windows Service running on an internal server. I will be sending the customer's POST data to ...
user avatar
5 votes
2 answers
1k views

How to handle thousands of messages to C socket?

I currently have a setup looking like this: __________________ _________ ________ ___________ | Front end server |----| Varnish |----| NodeJS |----| C-service | ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ...
user avatar
  • 151
2 votes
1 answer
3k views

Cache invalidation on multiple servers

I have 6 web servers which are giving me problems due to cache inconsistency. I am thinking of building a cache invalidation service such that there is a topic on which all the servers can publish a ...
user avatar
  • 123
2 votes
1 answer
2k views

Message persistence in Message Queues

I have looked into several message queues as referenced in this link. Several of the larger and more popular ones are in memory only. I can understand the need for this in some situations however in ...
user avatar
  • 121
4 votes
2 answers
739 views

Master-Slave Cluster - How to make sure the master is really dead for the slave to take over?

I have a in-house messaging system, similar to a message broker. We have one master message broker and one slave message broker. A message broker just receives messages and sends them to all nodes. ...
user avatar
3 votes
1 answer
126 views

Simple messaging pattern to replace shared DB? [closed]

There are two modules A and B using the same DB. A imports B as a JAR. I read that messaging patterns can replace the tightly coupled solution of using a shared DB but I am not sure which message ...
user avatar
  • 213
2 votes
2 answers
348 views

Eventing solutions for Java legacy applications too old for real JMS queue/topic eventing?

What are some architectural solutions that can mimic eventing for systems (Java) that are too old to implement eventing solutions, such as conventional Queue/Topic based JMS messaging? I have a ...
user avatar
19 votes
4 answers
8k views

Using flat files vs database/API as a transport between a frontend and backend

I've got an application which has generated a rather heated discussion between a couple of the developers. Basically, it's split into a web layer and a backend layer. The web layer collects ...
user avatar
1 vote
0 answers
369 views

Guaranteed message sending for messages of different priority for different modules

We have created a message que system using priority messages. The idea behind this is as follows: We have a list of clients. All clients know if (and what kind of priority) messages they have ...
user avatar
  • 365
3 votes
1 answer
3k views

What is an appropriate pattern for handling a queue of network events in C++?

I'm writing an application that sends and receives messages over the network and pushes them into a std::deque queue. and I'm looking for the appropriate programming pattern to handle all of the ...
user avatar
  • 133
6 votes
2 answers
4k views

Understanding the Single-Writer Principle

The article on the Single-Writer Principle of the Mechanical Sympathy blog explains how bad queues are (performance-wise), because they need to be able to receive messages from multiple produers, and ...
user avatar
  • 81
1 vote
2 answers
119 views

How to guarantee web script execution sequence/order or not parallels queues

This is a very simplified version of the problem: You have an integer saved on a server called "B" and you have an script saved on a server call "A" exposed to the internet. The script pulls the ...
user avatar
  • 109
4 votes
1 answer
2k views

Is it always the best way to go to queue emails instead of sending them real-time? [duplicate]

If you had to decide between queueing emails and sending them real-time, is it always the best way to go putting them in a queue and sending them one by one instead of instantly as soon as a user ...
user avatar
  • 235
2 votes
1 answer
121 views

Using short-lived REST requests to obtain messages from a message queue for clients

I would like to put together a system where clients obtain messages from a REST API, but with the producer being able to use the semantics of a message queue: e.g. ordering of messages, a destination ...
user avatar
  • 121
4 votes
3 answers
2k views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
user avatar
  • 317
2 votes
2 answers
4k views

Distributed transactions with Kafka

I need to implement a transaction that spans over loosely coupled (SOA+MOM) components. When a particular event is received, FOO and BAR need to do a transactional operation: that is, both the ...
user avatar
4 votes
1 answer
573 views

Azure ServiceBus Queue: Consumer Application

I am running a .net MVC4 web application. I want to be able to hit an action on a controller, create an email and send it to my service bus queue to be processed (sent). What are my options as far ...
user avatar
  • 141
5 votes
1 answer
1k views

Message queue vs database for delayed tasks

I need to build a system that can handle a fairly high amount of delayed tasks (e.g. scheduled emails). For non-delayed tasks I would go for something like RabbitMQ. But, is it ok to let tasks ...
user avatar
  • 53
2 votes
1 answer
790 views

How do I handle long running third party calls from backing up my message queue?

Here is my exact scenario. I must make requests to a third party service The service takes ~15 seconds to respond It also has no webhooks or any call back mechanisms These requests are not made ...
user avatar
  • 121
22 votes
1 answer
10k views

Does Akka obsolesce JMS/AMQP message brokers? [closed]

I spent the last week deep diving into the Akka docs and finally understand what actor systems are, and the problems that they solve. My understanding (and experience with) traditional JMS/AMQP ...
user avatar
  • 4,680
4 votes
1 answer
2k views

How many Topics (queues) to practically use in an ESB

In all the Service Bus examples and tutorials I've seen, they cover publishing a single message to a single Topic that is picked up by a single listening Subscriber. Whilst this is good for a simple ...
user avatar
4 votes
0 answers
1k views

Architecture design for Java servlet that triggers work (for sending push notifications)

I'm designing a push notification server (for pushing notification via APNS and Google servers) and have come up with the following design for a Java based web application server. I would really like ...
user avatar
  • 41
3 votes
2 answers
2k views

Is it worth it to use a message-queue framework in this case?

We have these things called "executions" that are run at certain times. They have a delay property that basically says "execute me after delay microseconds`. Currently we are using Cassandra as a ...
user avatar
1 vote
0 answers
104 views

Job queue dispatching based on resource constraints

We have a job-queue shaped hole in our architecture, but it isn't immediately obvious that there's a piece of existing software to fill it. There may be a reason for this, or there might not, so let ...
user avatar
5 votes
3 answers
3k views

Web Service and Message Queue

We're looking at creating a web services/REST API layer that will be consumed by web and mobile clients. To make the solution more robust I was considering putting the commands from PUT, POST, and ...
user avatar
2 votes
1 answer
122 views

Sequencing distributed tasks with a distributed DB

Background I have two separate processes, WriteIt() and ReadIt(). One creates records, and the other processes the records in a DB cluster. Once WriteIt() creates a record, it queues a ReadIt() task ...
user avatar
  • 123
1 vote
2 answers
230 views

Storm topology to handle "dating website"-like workloads

suppose im writing a dating website, similar to okcupid. there are profiles, and i need to compute the (N^2) "match" table - given every 2 profiles whats the match between them? I was thinking this ...
user avatar
  • 193
1 vote
1 answer
475 views

Options for repeated distributed task

Scenario My project has a need to periodically scan the database and notify administrators for particular issues. For example: Any errors in the logs since last update? Do values match expected ...
user avatar
  • 113
1 vote
3 answers
1k views

Using message queue systems - forcing synchronous processing

I have a php process producer-a which places a job on RabbitMQ a - deferring processing to a third-party service. The 3rd party service completes processing a and places a response onto queue b. I ...
user avatar
  • 1,007
5 votes
1 answer
5k views

Is having a bi directional message queue a design smell?

I have an architecture with 1 backend server and multiple frontend servers. The frontend servers are connected in a bi-directional (web sockets) connection to clients so they can send messages to ...
user avatar
1 vote
1 answer
385 views

Persist data downloaded by Celery workers

I'm working in a tool that downloads tweets from Twitter to process them later. For this purpose I'm using Celery with RabbitMQ, sending task with the keywords that must be tracked by the workers. My ...
user avatar
1 vote
0 answers
85 views

Improve communication between controller and trackers in a Twitter fetcher tool using RabbitMQ or Apache Flume

I've been working for a time with some researches developing a tool to fetch tweets from Twitter and process them in some way. The first prototype "worked" but became a pain as we used sockets to ...
user avatar
3 votes
3 answers
3k views

Design a Queuing Solution with Clustering and Multiple Consumers

It is a Design Problem which I am listing out here. I have different set of business operations that are carried out for different business entities. Operations: Operation A Operation B Operation ...
user avatar
1 vote
1 answer
102 views

More than 20 Records Inserted in a Span of 1 Sec

Well, this is not specific to any programming languages. I came across this situation in Android, iOS, PHP and .NET. The problem is many times I come across a situation while building IoT apps, where ...
user avatar
0 votes
0 answers
476 views

When to use a message queue as a web service database?

Does it make sense to implement a message queue system such as RabbitMQ or Kafka for the purpose of storing data in a queue? For example, this could be done in order to handle things such as ...
user avatar
  • 101
0 votes
1 answer
209 views

Should I have a queue with SMS requests to picked up by a separate process or connect directly to the API?

I created an email queue where I put the requests for emails to be sent. If the SMTP server is unavailable, the requests are not lost. When the SMTP server gets back online, the background service ...
user avatar
14 votes
3 answers
24k views

UML: Queue processor in a sequence diagram

Which is the appropriate way of representing a queue processor in a UML sequence diagram? I want to represent in the same diagram two systems that are loosely coupled through a queue. The first ...
user avatar
3 votes
1 answer
108 views

How to use Messagequeing in a distributed web application

I am curious for solutions to this problem: Assume we are going to build an online-store. To better scale parts of the architecture, it is broken into independend subsystems. The typical scenario ...
user avatar
  • 9,247
3 votes
1 answer
1k views

Passing notifications between programs written in different programming languages

I work in a team of people with experience in varied programming languages. Recently we were required to build a report when an order is placed in our system. Our ecommerce store runs on PHP and the ...
user avatar
  • 133
1 vote
1 answer
247 views

Locking database records

What I have right now: A table with lot of records. About 200 users can select a record according to some rules based on the data in each record. There is a high chance that 2 or more users will ...
user avatar
  • 6,528
5 votes
1 answer
231 views

How to Recover from Inconsistent Job State without Database Polling

I'm working on scaling an application which is currently polling a mySQL database to send async jobs to a background processing system. A state machine is being used to keep track of the entities ...
user avatar
  • 153
40 votes
1 answer
37k views

Why database as queue so bad? [closed]

I've just read this article, and I'm confused. Let's imagine 1 webapp and 1 distinct application acting as "worker", both sharing the same database. Oh, I said "sharing"..but what does the article ...
user avatar
  • 3,768
0 votes
1 answer
632 views

Estimating time remaining in a queue with multiple workers

I've been trying to figure out a way of calculating the estimated time remaining for an item in a queue when there are multiple workers processing items from the single queue. A client submits a job ...
user avatar
  • 253
3 votes
0 answers
369 views

How to schedule large numbers of user-defined tasks?

Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails). You could run a cron which SELECTs a database table for unprocessed ...
user avatar
4 votes
2 answers
2k views

Where does Windows Workflow Foundation fit into the messaging oriented middleware architecture?

I've been trying to find information online about where WWF 4.5 fits with in an architecture based around an enterprise message queue using central broker(s) (RabbitMQ in particular). It seems like ...
user avatar
  • 143
7 votes
1 answer
7k views

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

Current Scenario Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
user avatar
1 vote
1 answer
641 views

When does one NOT need to use queues in an application?

I've been developing an application using the Laravel Framework & have been exposed to the idea of queues for the first time. There are a multitude of tutorials & articles about using queues &...
user avatar
  • 21