Questions tagged [message-queue]

A message queue is a service that receives events or messages from producers and provides them to consumers.

Filter by
Sorted by
Tagged with
0 votes
1 answer
88 views

Message queuing systems: sequential processing with dependencies between queues

I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
user avatar
  • 109
-1 votes
1 answer
47 views

Any pattern for delaying processing of a (JMS) message until another task completes

Is there a good pattern in microservices systems for delaying the processing of a JMS message until some other task has been completed. Let's say, for example, a message is sent to a topic when a ...
user avatar
  • 179
0 votes
1 answer
63 views

Expose api to third parties to send PDF files (invoices)

Our system needs to process PDF invoice files from generic external third parties. Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads. Even ...
user avatar
  • 185
1 vote
2 answers
176 views

Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?

We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
user avatar
0 votes
1 answer
59 views

Prevent missing updates using a message broker after creating a task

I'm trying to design a data updates mechanism in my micro-services architecture. For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
user avatar
  • 109
1 vote
1 answer
118 views

Long-running compute-intensive tasks in APIs: background workers?

I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing. To give an impression of the compute requirements, ...
user avatar
  • 119
0 votes
0 answers
35 views

Design a Program which Forwards Work Pieces to Corresponding Modules

I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
user avatar
  • 1
0 votes
1 answer
76 views

A service having to have both an API and listen to events - Is this an anti-pattern for microservices?

I have a use case to create a service that consume messages from a message queue, process them, store them in the DB, and expose the processed results via an API. Therefore, the service I'm going to ...
user avatar
  • 109
1 vote
1 answer
441 views

Event bus vs PubSub

I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
user avatar
  • 119
0 votes
0 answers
66 views

Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?

I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
user avatar
  • 145
1 vote
1 answer
190 views

What is the common practice to serve a REST API based request asynchronously?

I am building a microservice app where the client sends requests through REST APIs to the backend server, and the backend will communicate with other services using gRPC. I am aware that this is ...
user avatar
0 votes
0 answers
42 views

design for the order processing system

We have REST endpoint that that works like this Retrieve payload from request Generated unique id for payload and order_status and enrichment_status to PENDING Store payload in Redis Perform request ...
user avatar
  • 203
1 vote
1 answer
401 views

Handling Errors in a Microservice architecture and asynchronous message queue

When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
user avatar
0 votes
2 answers
94 views

Scaled services and order of events sent as messages (using a broker)

How would you go about making sure that "events" sent as messages to any sort of broker (Kafka, RabbitMQ etc) from multiple instances of the same service (load balanced) are actually added ...
user avatar
  • 203
0 votes
1 answer
194 views

Removing successful cases from SQS batch. Is it wrong?

I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled. The most common problem with this kind of setup that I have seen is that, if ...
user avatar
2 votes
0 answers
230 views

System Design: Chat application in a distributed system context

I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
user avatar
-1 votes
2 answers
94 views

How to keep authentication in user context in reactive programming?

I want to execute everything in a user context. Meaning that if the request comes from one user then the service has a token of only this user and can't execute anything for another user. Now the ...
user avatar
0 votes
1 answer
71 views

Interactive and Batch traffic in one service [closed]

I am designing a workflow, and am trying to avoid parallel deployments of the same service. Thus I am looking to have one service that handles both interactive and batch traffic. My main concern is ...
user avatar
  • 109
0 votes
1 answer
123 views

What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?

I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic. How can ...
user avatar
1 vote
1 answer
85 views

Exposing redis to external clients

We are building a system that runs on our cloud and that needs information from our clients network that must not be exposed openly. We have concluded that the only way this could work, is if our ...
user avatar
  • 13
1 vote
0 answers
87 views

Retry long-running message processing in case of processing node failure

I have two types of microservices: Data Collector and Data Processor. They communicate with each other via Azure Service Bus Queue. Data Collector collects data from different sources and, once done, ...
user avatar
  • 149
0 votes
2 answers
71 views

Do I need a Message processing service before RabbitMQ (or any other message broker)

I hope that's a proper place to ask my question. I am wondering how is best to integrate RabbitMq (or any other message broker) into my project. There are 2 options: Simply collect all messages, ...
user avatar
  • 103
1 vote
2 answers
161 views

In Publish-subscribe pattern can a microservice publish message for itself

I am designing a somewhat complex flow in publish subscribe pattern using AWS SNS, SQS , Microservice and Lambda functions. To break the whole flow into multiple logical & smaller sub-flows I am ...
user avatar
1 vote
1 answer
345 views

Pattern to limit message queue processing based on external API throttling

I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
user avatar
  • 145
0 votes
0 answers
41 views

Architectural design for sending large amount of analytics data from production servers to s3 without impacting request performance

Lets say we have a server getting upto 1000 requests per second, serving them at p99 of 20ms (strong business case for not increasing this latency). The server gc parameters have been carefully tuned ...
user avatar
0 votes
0 answers
20 views

AWS Serverless Service to trigger an event when something doesn't happen

I'd like to use some sort of AWS serverless technology to trigger an event when a clickstream stops receiving clicks. Imagine a website visitor clicking around a site. Within our current solution ...
user avatar
-1 votes
1 answer
221 views

Handling multiple events in Event Driven Architecture

Let's say I have Service A that publishes Message 1 to be processed by Consumer A The same service A that publishes Message 2 to be processed by Consumer B I need to make sure that consumer B ...
user avatar
2 votes
1 answer
116 views

Implementing a persistent searchable "queue"

I'm receiving a lot of a data entries from different (IOT) devices. I need to temporarily and persistently store this data until it is polled by the matching clients, after which it will be deleted. ...
user avatar
  • 29
3 votes
0 answers
256 views

Implementation of projections in event-sourced system

I'm working on a application which uses event-sourcing and CQRS to define it's domain model. Background We have implemented projections to aggregate stream of all domain events into a read models used ...
user avatar
  • 35.6k
0 votes
1 answer
455 views

Replacing in-memory queue with messaging queue service

In our application (ASP.NET core) we have events that require computations and which results need to be persisted in a database (the objects are projects that need recalculations when a property ...
user avatar
1 vote
2 answers
582 views

Long polling and message brokers

I was reading this article, and the author uses HTTP long polling to design a chat application. I was wondering how does this compare with a dedicated Message Broker services like Apache Kafka, or ...
user avatar
  • 169
0 votes
0 answers
265 views

High availability RabbitMQ using gRPC as transport layer?

I have an idea to use gRPC for all microservices communication for the sake of simplicity. Now there is a service that needs high availability, So the idea is to use RabitMQ as a message receiver from ...
user avatar
  • 657
0 votes
1 answer
199 views

How can message queues improve scalability?

I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
user avatar
  • 145
1 vote
1 answer
114 views

Eventing: Determine if Set of Messages has been Processed

Given a Set of messages sent to a Queue, what are good ways to determine all of the messages have been processed? Constraints: Large number of messages Other messages besides those in the Set will be ...
user avatar
  • 109
0 votes
0 answers
260 views

gRPC client - to - client communication between other Lan network

I have a question about gRPC communication. I know gRPC has 4 models communication: unay, server-streaming, client-streaming and bi-direction. And the terms "client" and "server" ...
user avatar
3 votes
2 answers
138 views

Best approach for inter-process task queues

I have an application made up of multiple processes/workers/services which need to send messages to each other that represent units of "enqueued tasks" to be done. I am trying to find the ...
user avatar
  • 139
2 votes
1 answer
114 views

How (whether to?) include Apache Spark in my Architecture

Brief overview of general data flow The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic ...
user avatar
6 votes
2 answers
400 views

Design question for handling large volumes of messages in multi-tenant queue

I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based ...
user avatar
  • 169
-1 votes
1 answer
112 views

How to prevent races inside HTTP API?

You're writing the code for an API that accepts payloads from users and, before performing the actual write/update/delete in the db must validate the input for consistency and this is done by ...
user avatar
0 votes
0 answers
52 views

Architecture - For cloud services like GCP Cloud Run, should a restful API also subscribe to a message queue and how

In a bit of a quandary. I am using Cloud Run on GCP for backend services, and it will auto-scale based on HTTP load, potentially scaling down to 0 if there is no traffic for awhile. I have a product ...
user avatar
1 vote
1 answer
230 views

Distributed message queue, propagating queue leader/follower information

I'm designing a distributed queue message queue and I'm not sure how to get the leader information to another component. This is my current design: I have the following components: FrontEnd: request ...
user avatar
  • 249
1 vote
2 answers
187 views

How to make consumer idempotent when publisher publishes "at least once"?

I'm trying to implement microservices and I'm still super new to the subject, but, I was able to answer most of questions. However, one of the questions that I can't come up with a good solution to is ...
user avatar
1 vote
0 answers
300 views

Broker queue and topic naming conventions

I'm in the process of adding a message broker (ActiveMQ Artemis) to a system currently and while I have my own notions on how I can divide up the namespace, it seems like a good time to get opinions ...
user avatar
0 votes
3 answers
786 views

How to handle data inconsistency in microservice architecture?

Let's assume we have two services - user and auth with a message broker in between. user service handles CRUD action on user entity and auth service handles authentication. When a user is created/...
user avatar
2 votes
2 answers
353 views

Service bus and strict message order - how can I handle failed messages?

I am considering using Azure Service Bus, however my question applies to queues in general. I have a requirement to send messages in a certain order. Say that the publisher sends a message to the ...
user avatar
1 vote
0 answers
48 views

What is the recommended way to start a long-running containerized analysis in R from NodeJS?

I have a frontend through which users can submit analysis requests, which are inserted into a database. These analysis requests are received in a NodeJS server using Postgres notify/listen. Using ...
user avatar
0 votes
0 answers
226 views

Integrating HTTP / Webhooks with Message Queues

I'm working at a project which integrates several Applications mostly SaaS Applications. The SaaS solutions have all the possibilities to hook into the internal event system with webhooks. The ...
user avatar
  • 111
2 votes
1 answer
414 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
user avatar
  • 129
0 votes
0 answers
95 views

Is queue is the right choice for server request retries

I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. ...
user avatar
  • 109
0 votes
1 answer
71 views

Question about the best way to architect a small webapp

So I'm trying to create a simple website where I run a computationally intensive task on the backend based on input submitted by a form. For reference: my goal is not too dissimilar from something ...
user avatar
  • 109

1
2 3 4 5