I am trying to design a system that can be safely used to share messages between processes on the same system using ZeroMQ.
I have chosen to use a central broker that binds to a particular port. All clients would bind to the same port and publish to this using different topics.
- I have a network of multiple consumers and producers that talk to each other in a bi-directional flow. .i.e a consumer can occasionally act as a producer and vice versa.
- Given two services A&B, there are messages that are unique to A&B.
- Given services A,B...Z there are messages that are not unique to A&B .i.e there are messages that are consumed by A&B, but also by another service C.
Have central broker that connects to a specified port. All producers publish to this broker through this ZMQ context and subscribe from the same context. The socket type varies one is a PUB socket, other is a SUB socket. So essentially each service has one ZMQ context but two different sockets.
Service A&B can communicate to each other using the same broker, for unique messages I use a unique topic, lets say topic AB.
For non-unique messages consumed by multiple services, I have one topic say common.
My thought process
- Using a single broker makes it simpler, as opposed to this I would be looking at each service handling multiple ZMQ contexts, .i.e one context for each service it wants to talk to. Thus creating an overhead.
- Broker solves one problem of routing thus very important, but at the same time its dumb, and resilient.
- Unless the broker goes down, one service going down will not affect the other services.
Can the broker be made intelligent to monitor for crashes, queue when clients are down etc. I understand ZMQ the solves queuing at the subscriber level, hence freeing up the publisher, but with a broker, can it be done here? This is to add a layer of resilience at the broker level in case the clients go down.
Are the messages duplicated in this design?.
When looking at the common topic .i.e messages which are not unique to any service but are consumed by multiple services. Is there a way to add sub-topics / sub filtering?
Also, For communicating unique messages between services, would ZMQ push pull pattern help more than PUB SUB?
For example if there are 1000 messages on the common topic, I have sink that consumes to all messages.
Lets say a service C needs only 10 messages of these 1000, I don't want to filter all 1000 messages on the subscriber .i.e C, I would rather send only the 10 messages to C.
I don't see any way of doing this, also all my senses say this should not be technically possible as well. But want to know if there is any way to do it , I may have missed something.
The only way I see possible is:
- Send on different topics, on the sink, consume all relevant topics and multiplex them.
- Filter all messages on the common topic on each subscriber to get what each subscriber wants.