I'm building a messaging solution with the followingsetup: all the messages are saved on a DB, two or more reader processes will read from this DB and send data to other process(es) which will send it over the network.

My approach is depicted below,

The following have 4 sender process with 4 fifos, and 2 readers with 2 fifos

reader0 ← read data from DB
reader1 ← read data from DB

sending part
network_handler0 ← network_handler_fifo0 ← reader0
network_handler1 ← network_handler_fifo1 ← reader1
network_handler2 ← network_handler_fifo2 ← reader0
network_handler3 ← network_handler_fifo3 ← reader1

receiving part
network_handler0 → reader_fifo0 → reader0 → write to DB
network_handler1 → reader_fifo1 → reader1 → write to DB
network_handler2 → reader_fifo0 → reader0 → write to DB
network_handler3 → reader_fifo1 → reader1 → write to DB

I have few problem with this setup, and please note that the number of processes could be more than that based on the environment, so I could make it 20 readers and 10 network_handlers or it it could as shown above.

  1. The size of the buffer is 64K and the message size is 200k, is this small enough to make the write/read to/from fifo atomic?

  2. How can make the processes aware of each other, so for example, reader 0 writes to network_handler_fifo0 and network_handler_fifo2, how can I make it start writing on other fifo if the current ones are full or their network_handlers are dea d

  3. I thought about making the reader process writing more general in writing, so for example it writes to all network fifos using lock mechanism and stop writing on the one that its process dead, I didn't use it as lock mechanism could slow thing down.

BTW, each network_handler is an SCTP association, so network_handler0 is association 0, network_handler1 is association 1 and so on.

Any idea is appreciated. I mean even if I have to change the setup above.


Firstly, nothing is atomic in multi-processing situations. Do not expect to be able to ignore synchronisation just because you're using a relatively small payload. (even a 4-byte payload needs protection).

You might like to look at existing systems for this, I'd recommend ZeroMQ that offers you everything you need for this question.


It looks like your design might be a multithreaded client/server similar to the linked Code Project example. For the sake of efficiency, you might consider using a thread pool. You will need to protect your resources from multiple access even if you were dealing with just one byte, let alone a FIFO. I am not sure from your description whether this is a hardware or software FIFO, but I would expect you must control access to it on a per message basis.

Andrew Tannebaum used to recommend that you use queues for communication and processes for concurrency. Is there a reason that the reader and sender are two separate processes? Do you actually have separate processes or separate threads (i.e. the threads are part of a process and share memory)?

The classic Readers and Writers problem gives some advice on what to do with reads and writes for your database. The problem is not easy, but is explained in layers and the solution given on Wikipedia is worthy of emulation. One of my professors from grad school has a readers and writers page well, and shows multiple solutions. He gives a collection of pthreads based examples (including the consumer/producer problem).

Also included is a sem.c/.h wrapper library for pthreads that permits easier translation of example algorithms that use P() and V() that you might find sprinkled around the internet. P() and V() are semaphore functions as notated by Dutch computer scientist Edsger Dijkstra selected based on Dutch words as explained on Wikipedia. Sometimes semaphore functions are named signal() and wait(), which though clearer for English speakers, differs from what is used in a lot of research.

If you find that you have a dead process that you need to abandon, if you are using hardware FIFOs, it sounds like you lose them too. If a network resource goes down, that is something that definitely needs attention. Timeouts are kind of a bane of existence that makes sound, elegant algorithms into confused and unreliable implementations.

WRT the streaming aspects, somewhere in your algorithm you might want to drop packets in an evenly spaced fashion. I found a paper by researchers from MIT and Brown about streaming that might be of interest.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.