Is this an inefficient use of the Subscriber (PubSub) pattern?

Question

I have a Publisher object that raises an event at regular intervals (think of it something like a clock tick). I then have 100s (or 1000s) of subscriber objects (all instances of a single type or two) which may have to update themselves in response to the event. But most of the time the event won't apply to them, so they just ignore it. So the event handling method in the Subscriber looks something like this:

    private void changeHandler(int id)
    {
        if (id == this.ID) // rarely true
            val++; 
    }

I'm concerned because the inefficiency of this approach is starting to take a hit on performance. (Dozens of events raised every second, with thousands of subscribers, is becoming non-negligible.)

Now, it just so happens that the Publisher, in this case, does happen to know which subscribers each particular event applies to (which is how it can pass the id parameter), so I could bypass the Event strategy altogether and have the Publisher just update each subscriber "manually" (so to speak). But that would make the system more tightly coupled, which I was trying to avoid.

Is there a more efficient approach to this design strategy than I am currently using? Or a better pattern altogether that would maintain separation of concerns between the Publisher and the Subscribers?

Answer 1

The easiest solution would be to modify the subscribe function to take the id as an argument, and append the subscriber to a hashtable with the id as a key. Then the publish function would look up the id in the hashtable to get all the subscribers with that id, and publish only to those subscribers.

You would have some overhead above direct calls, but hashtable lookups are O(1), so it would certainly be a vast improvement over running an if statement on every potential subscriber.

Answer 2

Fundamentally, you can have the subscribers determine when to run, letting them see every publication. Or you can have the publisher "know" somehow what subscribers are interested in which events, creating a tighter coupling than we might want.

Or you can delegate this responsibility to a new third party, which might allow the decoupling you're looking for.

There's a logic algorithm, called the RETE Algorithm. This algorithm works with rules of the form: on condition do action. The conditions can be complex expressions.

The algorithm seeks to optimize the testing of the conditions for each rule in response to events, in cases where there could be thousands upon thousands of rules. What the RETE algorithm does is like common sub-expression detection and elimination in compiler technology. It finds common sub-expressions in the various rules' condition part and creates an alternate, internal rule structure that represents the common sub conditions across all the rules each only once. It maintains a this data structure such that the actual rules can be dynamically add/removed/changed. It allows for complex event conditions (i.e. a series of events): in its internal rule-condition structure, it remembers which sub expressions ones are already activated (already match from prior events). Thus, since it has precomputed both the sub-expressions and certain prior conditions, when an event does occur, it doesn't have to do that much work to advance the state of the internal structure, and eventually fire a rule's actions.

There is a parallel here, where subscribers have a condition on which they will can act, and there may be many, many subscribers, and they are all observing similar attributes; further, subscribers may change their conditions dynamically.

---

Beyond that, there is nothing to stop the subscribers from still doing their own condition testing and doing nothing if there isn't a proper match. This means you could simplify the (expression language of the) third party in some cases, so there some potential to balance the third party complexity vs. the inefficiency of notifying subscribers that wouldn't do anything anyway.

Answer 3

Your publisher is already tightly coupled to each subscriber, since it knows subscriber identities and expects them to be up and running. As both you and Karl Bielefeldt observe, having N distinct pubsub channels for N subscribers would make sense. In apache kafka you could use ID directly as the channel name.

If you're looking for resiliency and decoupling, you could use 0.5 * N channels, and require the recipient to acknowledge with an ACK on the channel. That way his buddy would notice missing ACKs and could assume responsibility. This is a K=2 buddy group, but larger K would work, also.

Your example of incrementing a local counter was very simple, perhaps for illustrative purposes. Counter increment events can be conveniently batched up. If up-to-the-moment consistency is not crucial to your application, the publisher might want to aggregate some of those messages to reduce message traffic.