I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory architecture.

Does the producer-consumer problem appear in communication between processes in distributed memory architecture? If yes,

  • Is the output generated by a producer stored in a place shared with the consumer?

  • Do the same synchronization methods (e.g. locks, semaphore, monitor, ...) used in shared memory architecture apply to the producer-consumer problem in distributed memory architecture?

I saw "pipeline" and "stream processing" are popular words. Do they mean the producer-consumer problem/pattern? Do they require the same synchronization methods as in shared memory architecture?



You are right in thinking that the problems of races and deadlocks in producer-consumer stem from shared variables and data structures.

Any system that allows multiple accessors (processes, threads, interrupt handler & main, sometimes even simple recursion, or object patterns) and shared variables or shared data structures featuring raw read and write access can have such problems.

For example, even a distributed system in which:

  • A shared database (e.g. SQL, key-value, etc..) is used to store, read, and write variables or a data structure, non-atomically — if one actor reads the database, and as a result of testing some value, a suspend message is sent to the other actor (who can also update variables in the database) we'll have the same problems as with shared memory.

  • A shared file system used to store same variables, updated by multiple actors, can suffer the same problems.

We need to use domain-oriented abstractions that manipulate data structures, rather than raw read & write capabilities performed by independent and unsynchronized actors.

When I think of distributed systems; however, I think of meaningful, high-level message passing — message passing systems can be designed to send messages about job data to be worked on rather than messages managing shared variables.

To be clear, a node in a distributed system internally has to be able to buffer network packets, which means enqueue and dequeue, and that is a producer-consumer arrangement; so, of course, these implementations will protect internally shared buffers using locks or other, but this queuing is typically done at the system level below that of regular user code.

In a distributed system we expect a consumer to naturally suspend itself when it has processed all (buffered) job packets, and to resume from suspension on arrival of a new packet — thus, the producer doesn't necessarily have to wake a consumer at all; it just sends a job packet.  This naturally takes care of the empty buffer situation in producer-consumer.  By comparison with the producer-consumer approach given at wikipedia, this breaks the cyclic nature of the actors woken graph (consumer wakes producer though also producer wakes consumer).

A robust distributed system would also offer some back pressure mechanism, which would have a consumer send throttling messages (suspend/resume) to the producer — otherwise a producer producing faster than a consumer consumes would eventually overwhelm the consumer.  This is similar to the full buffer condition of producer-consumer; however, by comparison, does not involve shared variables.

As with shared memory-based producer-consumer, multiple producers or consumers complicate matters in a distributed system, so there would likely be a coordinating service that manages the job queue — though not necessarily through external manipulation of shared variables.  (Such a coordinating service could also facilitate retry on network and worker failures.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.