Questions tagged [producer-consumer]
The producer-consumer tag has no usage guidance.
27 questions
0
votes
1
answer
52
views
Which techniques to follow to make a fast and consistent message consumer that writes to a RDBMS [closed]
I am building a consumer application that reads messages from a queue and runs basic sanitization and deduplication and persists them to an RDBMS.
The functional requirements are
messages must not be ...
- 131
1
vote
1
answer
425
views
What kind of queue should I use for processing large volume of data?
I have a segment of customers which may range between 1 and 5000. For each customers, I need to do some processing by making some query to the database using the data from the customer. The query is ...
- 121
4
votes
1
answer
2k
views
One producer - multiple consumers queue, best way to guard against rare duplicates?
I have a classical one producer - multiple consumers queue, each item can be processed independently. On rare occasions (<1%) there can be multiple queue items related to the same object the ...
0
votes
0
answers
49
views
Architectural design for sending large amount of analytics data from production servers to s3 without impacting request performance
Lets say we have a server getting upto 1000 requests per second, serving them at p99 of 20ms (strong business case for not increasing this latency). The server gc parameters have been carefully tuned ...
- 109
3
votes
1
answer
4k
views
What is the currently best C# implementation for a Producer/Consumer scenario?
I have a relatively simple task where I need some 10 consumers to consume work to be produced into a queue, continuously.
This is my first time implementing this design pattern, so I have been ...
- 471
1
vote
1
answer
293
views
Implemeting a distributed n-Producers/1-Consumer service for critical-mission system
I am trying to implement a distributed version of multi-producer/1-consumer for a critical-mission system. I'm looking for good alternatives to the current approach based on RDBMS.
The problem
The ...
- 113
1
vote
1
answer
46
views
optimal way of dealing with subscriptions
I have a system in which some data is inserted into database. Whenever that data is inserted subscribers should be notified about that (it doesn't have to happen just after inserting, but there is a ...
- 111
1
vote
1
answer
828
views
Does the producer-consumer problem appear in both shared memory and distributed memory architectures?
I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory ...
- 5,495
5
votes
2
answers
950
views
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says
Batch processing systems (offline systems) Chapter 10
A batch processing system takes a large amount of input data, runs a
job to pro‐ cess it, and ...
- 5,495
5
votes
2
answers
2k
views
kafka consumer properties session.timeout.ms vs heartbeat.interval.ms
I am reading Kafka, the definitive guide and I came across the below point for consumers.
heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout ...
- 59
1
vote
0
answers
28
views
Consumer triggered entity data agregation
I am currently using RabbitMQ to aggregate the output of various data sources.
Functionally the process is the following:
The user enters a product name.
The request is sent to the different sources ...
- 119
0
votes
0
answers
83
views
How to send messages in the order they were queued, while ensuring that client B does not have to wait until client A has received his message?
I have a simplified producer/consumer pattern implemented below. The code outputs:
"A"
1 second delay
"B"
1 second delay
"A"
1 second delay
"B"
...
What ...
- 39
0
votes
2
answers
1k
views
Best design for a AMQP work queue
I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will ...
- 1
2
votes
3
answers
4k
views
How to build rest calls dynamically to interface with different APIs?
Lets say I am building an application where clients can book bus tickets. The only thing they do is select, origin, destination and the type of service (express or standard). The system will then book ...
- 327
1
vote
1
answer
369
views
Is my approach at building middleware using a queue a good approach?
I want to build a middleware system that my company can use and expand on to handle the integration points between our various bespoke and 3rd party systems as well as our hosted websites.
I am ...
- 369
2
votes
2
answers
4k
views
Solve Synchronization
I want to solve Synchronization with a Message Queue
In the wikipedia page of message queue, I read:
Most messaging systems support both the publisher/subscriber and message queue models in their ...
- 180
3
votes
2
answers
208
views
Is it a good practice if using Object when a method can accept more than one type of object?
public class TaskA implements Runnable {
....
}
public class TaskB implements Runnable {
....
}
I have two runnable class as TaskA, TaskB. I want to have a TaskManager to run these two tasks and have ...
- 79
5
votes
1
answer
24k
views
Parsing Multiple Files and Their Contents in Java using Multithreading without ExecutorService
I'm learning concurrency in Java and went over the tutorials on Oracle website. While I have understood some of it, a greater portion eludes me. I was thinking of a hypothetical problem (though it may ...
- 163
3
votes
1
answer
3k
views
Producer/consumer problem with 2 mutexes?
The setup
Let us consider a variant of the classical multithreaded programming problem: the producer/consumer problem. We have some finite number of resources (for simplicity, let there be just a ...
- 139
3
votes
1
answer
427
views
How to recognize consumers in AMQP architecture
I'm doing an project where I'm using a microservice architecture. All the services within this architecture communicate with each other using an AMQP broker (RabbitMQ).
When something happens in a ...
- 191
1
vote
0
answers
143
views
Trade off between concurrency and stagnation in a Dining Philosophers problem analogue?
I am currently facing a problem in the application I am developing that I think is the same as, or an analogue of, the Dining Philosophers Problem.
In my application I have a list of N resources. I ...
- 185
1
vote
1
answer
702
views
Producer and multiple consumers with limited types item
I am implementing producer-consumer where items to be consumed are of 1000 types type-0, type-1, ..., type-999.
Producer inserts item of type-i in queue. Consumer uses that to update state machine ...
- 141
6
votes
2
answers
5k
views
Understanding the Single-Writer Principle
The article on the Single-Writer Principle of the Mechanical Sympathy blog explains how bad queues are (performance-wise), because they need to be able to receive messages from multiple produers, and ...
- 81
17
votes
1
answer
13k
views
Difference between Consumer/Producer and Observer/Observable
I am working on the design of an application that consists of three parts:
a single thread that watches for certain events happening (file creation, external requests etc.)
N worker threads that ...
- 173
3
votes
0
answers
873
views
An efficient way of starting an arbitrary number of consumer threads?
I have a Producer/Consumer implementation where the number of consumers is configurable (this is a form of configurable throttling). The producer and consumer are kicked off like this:
var cts = new ...
- 131
3
votes
0
answers
403
views
How to schedule large numbers of user-defined tasks?
Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails).
You could run a cron which SELECTs a database table for unprocessed ...
- 173
33
votes
3
answers
34k
views
How to implement a message queue over Redis?
Why Redis for queuing?
I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...
- 2,212