1

I have a Publisher object that raises an event at regular intervals (think of it something like a clock tick). I then have 100s (or 1000s) of subscriber objects (all instances of a single type or two) which may have to update themselves in response to the event. But most of the time the event won't apply to them, so they just ignore it. So the event handling method in the Subscriber looks something like this:

    private void changeHandler(int id)
    {
        if (id == this.ID) // rarely true
            val++; 
    }

I'm concerned because the inefficiency of this approach is starting to take a hit on performance. (Dozens of events raised every second, with thousands of subscribers, is becoming non-negligible.)

Now, it just so happens that the Publisher, in this case, does happen to know which subscribers each particular event applies to (which is how it can pass the id parameter), so I could bypass the Event strategy altogether and have the Publisher just update each subscriber "manually" (so to speak). But that would make the system more tightly coupled, which I was trying to avoid.

Is there a more efficient approach to this design strategy than I am currently using? Or a better pattern altogether that would maintain separation of concerns between the Publisher and the Subscribers?

|improve this question
  • Do/can you know statically which subscribers are interested in which events? – Erik Eidt Sep 7 '17 at 22:42
  • @ErikEidt: no, that can only be known at runtime. – kmote Sep 8 '17 at 2:12
  • 1
    In a broker-based messaging system, a broker acts as a 'middleman' capable of filtering and routing based on data in a message header before forwarding messages to interested subscribers. Publishers send a message to the broker once, specifying various bits of 'routing' information (such as a subscriber id, message type, message category, routing key, and/or other discriminators where necessary) in the header; the broker maintains a data structure used to decide which messages are forwarded to which subscribers. Publishers and Subscribers are completely disconnected from each other. – Ben Cottrell Sep 8 '17 at 6:10
2

The easiest solution would be to modify the subscribe function to take the id as an argument, and append the subscriber to a hashtable with the id as a key. Then the publish function would look up the id in the hashtable to get all the subscribers with that id, and publish only to those subscribers.

You would have some overhead above direct calls, but hashtable lookups are O(1), so it would certainly be a vast improvement over running an if statement on every potential subscriber.

|improve this answer
1

Fundamentally, you can have the subscribers determine when to run, letting them see every publication. Or you can have the publisher "know" somehow what subscribers are interested in which events, creating a tighter coupling than we might want.

Or you can delegate this responsibility to a new third party, which might allow the decoupling you're looking for.

There's a logic algorithm, called the RETE Algorithm. This algorithm works with rules of the form: on condition do action. The conditions can be complex expressions.

The algorithm seeks to optimize the testing of the conditions for each rule in response to events, in cases where there could be thousands upon thousands of rules. What the RETE algorithm does is like common sub-expression detection and elimination in compiler technology. It finds common sub-expressions in the various rules' condition part and creates an alternate, internal rule structure that represents the common sub conditions across all the rules each only once. It maintains a this data structure such that the actual rules can be dynamically add/removed/changed. It allows for complex event conditions (i.e. a series of events): in its internal rule-condition structure, it remembers which sub expressions ones are already activated (already match from prior events). Thus, since it has precomputed both the sub-expressions and certain prior conditions, when an event does occur, it doesn't have to do that much work to advance the state of the internal structure, and eventually fire a rule's actions.

There is a parallel here, where subscribers have a condition on which they will can act, and there may be many, many subscribers, and they are all observing similar attributes; further, subscribers may change their conditions dynamically.

Beyond that, there is nothing to stop the subscribers from still doing their own condition testing and doing nothing if there isn't a proper match. This means you could simplify the (expression language of the) third party in some cases, so there some potential to balance the third party complexity vs. the inefficiency of notifying subscribers that wouldn't do anything anyway.

|improve this answer
  • Thank you for the interesting suggestion, but in this case I'm afraid it won't be of much help. The inefficiency in my scenario has nothing to do with complex rules. (The subscribers in my tool are essentially just loggers waiting for random triggers.) – kmote Sep 8 '17 at 2:18
  • @kmote Erik Eidt's point is that you can 1) do what you were doing, 2) do what Karl Bielefeldt suggests, or 3) you could use a third object as a mediator. The RETE algorithm is just a very elaborate example of 3. As a much simpler example, you can keep the publisher code as you have it, i.e. just notify anyone who's listening, and have it so the subscribers are only notified about things they are interested in by having a "dispatcher" intermediary do the hashtable lookup described in Karl Bielefeldt's answer. – Derek Elkins Sep 8 '17 at 21:40
1

Your publisher is already tightly coupled to each subscriber, since it knows subscriber identities and expects them to be up and running. As both you and Karl Bielefeldt observe, having N distinct pubsub channels for N subscribers would make sense. In apache kafka you could use ID directly as the channel name.

If you're looking for resiliency and decoupling, you could use 0.5 * N channels, and require the recipient to acknowledge with an ACK on the channel. That way his buddy would notice missing ACKs and could assume responsibility. This is a K=2 buddy group, but larger K would work, also.

Your example of incrementing a local counter was very simple, perhaps for illustrative purposes. Counter increment events can be conveniently batched up. If up-to-the-moment consistency is not crucial to your application, the publisher might want to aggregate some of those messages to reduce message traffic.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.