Questions tagged [message-queue]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-1
votes
0answers
16 views

Do you need a message bus for a SSR React app with a SQL Database?

I've been wanting to use the Remix React framework and was thinking of how to scale it. In Remix, it seems you have loader functions that load the data the page will need so that it can render ...
0
votes
0answers
57 views

Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?

I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
1
vote
1answer
148 views

What is the common practice to serve a REST API based request asynchronously?

I am building a microservice app where the client sends requests through REST APIs to the backend server, and the backend will communicate with other services using gRPC. I am aware that this is ...
0
votes
0answers
36 views

design for the order processing system

We have REST endpoint that that works like this Retrieve payload from request Generated unique id for payload and order_status and enrichment_status to PENDING Store payload in Redis Perform request ...
1
vote
1answer
113 views

Handling Errors in a Microservice architecture and asynchronous message queue

When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
0
votes
2answers
89 views

Scaled services and order of events sent as messages (using a broker)

How would you go about making sure that "events" sent as messages to any sort of broker (Kafka, RabbitMQ etc) from multiple instances of the same service (load balanced) are actually added ...
0
votes
1answer
66 views

Removing successful cases from SQS batch. Is it wrong?

I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled. The most common problem with this kind of setup that I have seen is that, if ...
1
vote
0answers
89 views

System Design: Chat application in a distributed system context

I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
0
votes
2answers
63 views

How to keep authentication in user context in reactive programming?

I want to execute everything in a user context. Meaning that if the request comes from one user then the service has a token of only this user and can't execute anything for another user. Now the ...
0
votes
1answer
69 views

Interactive and Batch traffic in one service [closed]

I am designing a workflow, and am trying to avoid parallel deployments of the same service. Thus I am looking to have one service that handles both interactive and batch traffic. My main concern is ...
0
votes
1answer
71 views

What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?

I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic. How can ...
1
vote
1answer
56 views

Exposing redis to external clients

We are building a system that runs on our cloud and that needs information from our clients network that must not be exposed openly. We have concluded that the only way this could work, is if our ...
1
vote
0answers
85 views

Retry long-running message processing in case of processing node failure

I have two types of microservices: Data Collector and Data Processor. They communicate with each other via Azure Service Bus Queue. Data Collector collects data from different sources and, once done, ...
0
votes
2answers
61 views

Do I need a Message processing service before RabbitMQ (or any other message broker)

I hope that's a proper place to ask my question. I am wondering how is best to integrate RabbitMq (or any other message broker) into my project. There are 2 options: Simply collect all messages, ...
1
vote
2answers
111 views

In Publish-subscribe pattern can a microservice publish message for itself

I am designing a somewhat complex flow in publish subscribe pattern using AWS SNS, SQS , Microservice and Lambda functions. To break the whole flow into multiple logical & smaller sub-flows I am ...
1
vote
1answer
114 views

Pattern to limit message queue processing based on external API throttling

I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
0
votes
0answers
40 views

Architectural design for sending large amount of analytics data from production servers to s3 without impacting request performance

Lets say we have a server getting upto 1000 requests per second, serving them at p99 of 20ms (strong business case for not increasing this latency). The server gc parameters have been carefully tuned ...
0
votes
0answers
18 views

AWS Serverless Service to trigger an event when something doesn't happen

I'd like to use some sort of AWS serverless technology to trigger an event when a clickstream stops receiving clicks. Imagine a website visitor clicking around a site. Within our current solution ...
-1
votes
1answer
118 views

Handling multiple events in Event Driven Architecture

Let's say I have Service A that publishes Message 1 to be processed by Consumer A The same service A that publishes Message 2 to be processed by Consumer B I need to make sure that consumer B ...
2
votes
1answer
76 views

Implementing a persistent searchable "queue"

I'm receiving a lot of a data entries from different (IOT) devices. I need to temporarily and persistently store this data until it is polled by the matching clients, after which it will be deleted. ...
3
votes
0answers
204 views

Implementation of projections in event-sourced system

I'm working on a application which uses event-sourcing and CQRS to define it's domain model. Background We have implemented projections to aggregate stream of all domain events into a read models used ...
0
votes
1answer
238 views

Replacing in-memory queue with messaging queue service

In our application (ASP.NET core) we have events that require computations and which results need to be persisted in a database (the objects are projects that need recalculations when a property ...
0
votes
1answer
284 views

Long polling and message brokers

I was reading this article, and the author uses HTTP long polling to design a chat application. I was wondering how does this compare with a dedicated Message Broker services like Apache Kafka, or ...
0
votes
0answers
125 views

High availability RabbitMQ using gRPC as transport layer?

I have an idea to use gRPC for all microservices communication for the sake of simplicity. Now there is a service that needs high availability, So the idea is to use RabitMQ as a message receiver from ...
0
votes
1answer
89 views

How can message queues improve scalability?

I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
1
vote
1answer
94 views

Eventing: Determine if Set of Messages has been Processed

Given a Set of messages sent to a Queue, what are good ways to determine all of the messages have been processed? Constraints: Large number of messages Other messages besides those in the Set will be ...
0
votes
0answers
121 views

gRPC client - to - client communication between other Lan network

I have a question about gRPC communication. I know gRPC has 4 models communication: unay, server-streaming, client-streaming and bi-direction. And the terms "client" and "server" ...
3
votes
2answers
119 views

Best approach for inter-process task queues

I have an application made up of multiple processes/workers/services which need to send messages to each other that represent units of "enqueued tasks" to be done. I am trying to find the ...
2
votes
1answer
78 views

How (whether to?) include Apache Spark in my Architecture

Brief overview of general data flow The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic ...
5
votes
2answers
281 views

Design question for handling large volumes of messages in multi-tenant queue

I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based ...
-1
votes
1answer
112 views

How to prevent races inside HTTP API?

You're writing the code for an API that accepts payloads from users and, before performing the actual write/update/delete in the db must validate the input for consistency and this is done by ...
0
votes
0answers
44 views

Architecture - For cloud services like GCP Cloud Run, should a restful API also subscribe to a message queue and how

In a bit of a quandary. I am using Cloud Run on GCP for backend services, and it will auto-scale based on HTTP load, potentially scaling down to 0 if there is no traffic for awhile. I have a product ...
1
vote
1answer
178 views

Distributed message queue, propagating queue leader/follower information

I'm designing a distributed queue message queue and I'm not sure how to get the leader information to another component. This is my current design: I have the following components: FrontEnd: request ...
1
vote
2answers
142 views

How to make consumer idempotent when publisher publishes "at least once"?

I'm trying to implement microservices and I'm still super new to the subject, but, I was able to answer most of questions. However, one of the questions that I can't come up with a good solution to is ...
1
vote
0answers
199 views

Broker queue and topic naming conventions

I'm in the process of adding a message broker (ActiveMQ Artemis) to a system currently and while I have my own notions on how I can divide up the namespace, it seems like a good time to get opinions ...
0
votes
3answers
572 views

How to handle data inconsistency in microservice architecture?

Let's assume we have two services - user and auth with a message broker in between. user service handles CRUD action on user entity and auth service handles authentication. When a user is created/...
1
vote
2answers
233 views

Service bus and strict message order - how can I handle failed messages?

I am considering using Azure Service Bus, however my question applies to queues in general. I have a requirement to send messages in a certain order. Say that the publisher sends a message to the ...
1
vote
0answers
35 views

What is the recommended way to start a long-running containerized analysis in R from NodeJS?

I have a frontend through which users can submit analysis requests, which are inserted into a database. These analysis requests are received in a NodeJS server using Postgres notify/listen. Using ...
0
votes
0answers
189 views

Integrating HTTP / Webhooks with Message Queues

I'm working at a project which integrates several Applications mostly SaaS Applications. The SaaS solutions have all the possibilities to hook into the internal event system with webhooks. The ...
2
votes
1answer
330 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
0
votes
0answers
87 views

Is queue is the right choice for server request retries

I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. ...
0
votes
1answer
66 views

Question about the best way to architect a small webapp

So I'm trying to create a simple website where I run a computationally intensive task on the backend based on input submitted by a form. For reference: my goal is not too dissimilar from something ...
1
vote
1answer
35 views

optimal way of dealing with subscriptions

I have a system in which some data is inserted into database. Whenever that data is inserted subscribers should be notified about that (it doesn't have to happen just after inserting, but there is a ...
1
vote
1answer
167 views

Designing Kafka topics for secured event driven job scheduling system

Consider the following Group A Job A { Depends on Job B of Group A Run User -> User1 } Job B { Depends on Job C and Job D of Group A Run User -> User2 } Job C { Depends ...
0
votes
1answer
65 views

Process AMPQ messages both reliably and fast

I am new to RabbitMQ and herewith I want to make sure that I am not missing out on some advanced RabbitMQ feature or pattern I am not aware of. I need to develop a reliable system that processes a ...
0
votes
1answer
150 views

Integrating with real-time data from multiple devices with accelerometer

I am designing a MVP for a simple gamification system for a trampoline park(s). External company is providing bracelets for customers, then collecting this data in their own web app, where they ...
4
votes
2answers
1k views

Communicating bulk data among microservices

Is there a pattern or design that I could refer to for dealing with bulk data in inter-services communication. My use case is to import data from upstream feed files(say 50k records) to our ...
0
votes
3answers
131 views

Keeping up logical order in a messaging environment

My setting is the following (Please assume that this points are entirely rock solid unchangeable, some for good reasons some: "just because"): In a super scalable microservice environment i receive ...
0
votes
2answers
346 views

Moving single tenant application with queue to multi tenants web application

I need to move a single-tenant web application to a multi-tenant (about 100 tenants) web application. Tenants are going to share the same application but each tenant is going to have its own database (...
1
vote
1answer
119 views

Alternative architecture for multi-instance, client-to-client message relay/broker

Research leads me to believe the most common architecture for a multi-instance (horizontally-scaling) message relay service (the typical example being an instant messenger/chat service) is to ...

1
2 3 4 5