Message passing interface between protocol layers

Question

Let's imagine a custom network application having 3 protocol layers.

Layer A is the top-level protocol and layer C is the lower level protocol. Each layer has its distinct functionalities. These layers need to communicate among them. To my understanding, we can simply call methods of protocol layer B from A's methods.

Recently I got to know that, this layered communication can be implemented as different entities (threads) and they wait on a specific queue to receive messages from other layers. In C++ terms, I can think of using std::queue, std::thread and synchronize read and write to the queue using mutex and conditional variables.

Before implementing this message queue mechanism, I would like to know, is there any abstraction level (key consideration) that I am missing? I don't need any C++ specific implementation details, but if you provide, that's a plus for me.

Answer 1

It seems that you have already well thought about your design. I do not see something fundamental missing. But if deemed useful, here some further thoughts.

If I understand well your intent:

• you’ll have three threads TA, TB, TC
• each thread will run an instance of a specialized class acting as protocol handler PA, PB, PC
• the input messages for each protocol handler shall be queued. So you’ll have the queues QA, QB, QC.

It is important to note that a thread does not run an object, but executes code to whatever object it belongs. This is not a problem in your case, since you want to isolate PA, PB and PC from each other, by channelling their interactions through queues instead of direct function calls.

The consequence is that each queue will be touched by several threads. So there is a (high) risk of race condition, when for example one thread writes an element into the queue and at the same time another thread reads the information. Standard library queues are not thread-safe.

To avoid the race you could simply lock a queue when it is accessed. But this might make your queue the bottleneck. Instead, you could consider using a lock free queue to reduce contention (you should use an existing implementation like boost’s one; or the one proposed in Anthony Williams book “C++ concurrency in action”).

Another thing to consider is that you could in addition to the vertical split into threads also split horizontally. So there could be more than one thread dedicated to the same layer. So you could have TA1 and TA2 running PA1 and PA2. All the clones would just have to listen to the same queue QA.

To allow this design and keep it flexible, the reference to the queues to use (input queue, downward and upward output queues) could be provided to the constructor of the protocol handler.