Questions tagged [producer-consumer]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
55 views

Does the producer-consumer problem appear in both shared memory and distributed memory architectures?

I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory ...
4
votes
2answers
91 views

What are the distinction and relation between batch processing and stream processing systems?

Design Data Intensive Applications says Batch processing systems (offline systems) Chapter 10 A batch processing system takes a large amount of input data, runs a job to pro‐ cess it, and ...
1
vote
0answers
101 views

kafka consumer properties session.timeout.ms vs heartbeat.interval.ms

I am reading Kafka, the definitive guide and I came across the below point for consumers. heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout ...
1
vote
0answers
22 views

Consumer triggered entity data agregation

I am currently using RabbitMQ to aggregate the output of various data sources. Functionally the process is the following: The user enters a product name. The request is sent to the different sources ...
0
votes
0answers
62 views

How to send messages in the order they were queued, while ensuring that client B does not have to wait until client A has received his message?

I have a simplified producer/consumer pattern implemented below. The code outputs: "A" 1 second delay "B" 1 second delay "A" 1 second delay "B" ... What ...
0
votes
2answers
156 views

Best design for a AMQP work queue

I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will ...
2
votes
3answers
421 views

How to build rest calls dynamically to interface with different APIs?

Lets say I am building an application where clients can book bus tickets. The only thing they do is select, origin, destination and the type of service (express or standard). The system will then book ...
1
vote
1answer
261 views

Is my approach at building middleware using a queue a good approach?

I want to build a middleware system that my company can use and expand on to handle the integration points between our various bespoke and 3rd party systems as well as our hosted websites. I am ...
2
votes
2answers
2k views

Solve Synchronization

I want to solve Synchronization with a Message Queue In the wikipedia page of message queue, I read: Most messaging systems support both the publisher/subscriber and message queue models in their ...
3
votes
2answers
193 views

Is it a good practice if using Object when a method can accept more than one type of object?

public class TaskA implements Runnable { .... } public class TaskB implements Runnable { .... } I have two runnable class as TaskA, TaskB. I want to have a TaskManager to run these two tasks and have ...
3
votes
1answer
16k views

Parsing Multiple Files and Their Contents in Java using Multithreading without ExecutorService

I'm learning concurrency in Java and went over the tutorials on Oracle website. While I have understood some of it, a greater portion eludes me. I was thinking of a hypothetical problem (though it may ...
3
votes
1answer
2k views

Producer/consumer problem with 2 mutexes?

The setup Let us consider a variant of the classical multithreaded programming problem: the producer/consumer problem. We have some finite number of resources (for simplicity, let there be just a ...
3
votes
1answer
385 views

How to recognize consumers in AMQP architecture

I'm doing an project where I'm using a microservice architecture. All the services within this architecture communicate with each other using an AMQP broker (RabbitMQ). When something happens in a ...
1
vote
0answers
116 views

Trade off between concurrency and stagnation in a Dining Philosophers problem analogue?

I am currently facing a problem in the application I am developing that I think is the same as, or an analogue of, the Dining Philosophers Problem. In my application I have a list of N resources. I ...
1
vote
1answer
539 views

Producer and multiple consumers with limited types item

I am implementing producer-consumer where items to be consumed are of 1000 types type-0, type-1, ..., type-999. Producer inserts item of type-i in queue. Consumer uses that to update state machine ...
6
votes
2answers
2k views

Understanding the Single-Writer Principle

The article on the Single-Writer Principle of the Mechanical Sympathy blog explains how bad queues are (performance-wise), because they need to be able to receive messages from multiple produers, and ...
16
votes
1answer
10k views

Difference between Consumer/Producer and Observer/Observable

I am working on the design of an application that consists of three parts: a single thread that watches for certain events happening (file creation, external requests etc.) N worker threads that ...
3
votes
0answers
831 views

An efficient way of starting an arbitrary number of consumer threads?

I have a Producer/Consumer implementation where the number of consumers is configurable (this is a form of configurable throttling). The producer and consumer are kicked off like this: var cts = new ...
3
votes
0answers
333 views

How to schedule large numbers of user-defined tasks?

Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails). You could run a cron which SELECTs a database table for unprocessed ...
29
votes
3answers
24k views

How to implement a message queue over Redis?

Why Redis for queuing? I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...