Use of Service Bus in a Pub-Sub Engine

Question

In one of our projects, we've built a Publisher - Subscriber Engine on Oracle Service Bus. The functionality being a series of events are published and subscribers (JMS queues) receive these whenever a new event is published.

We are facing some technical issues now, performance-wise and hence an architectural review is underway.

Now for my questions:

Architecturally the ESB has to publish events into a DB and read from the DB which users wish to be notified, then push the event onto their respective queues.

There is a high amount of DB interaction and the question is whether ESB should be having such high amount of interaction with the DB in the first place? Or should there have been some alternate component responsible for doing this.

Alternately is there any non-DB approach in which we can store the events and subscribers? Where else can this application data be held within the ESB context?

Answer 1

Have you considered event-driven systems, such as Complex Event Processing engines? These are tools designed to handle high volume streaming events. Some support integration with a database for historical correlation and data storage.

A prime example of an open-source CEP engine is Esper, although I believe database integration is part of the commercial Esper Enterprise solution. I haven't used this particular tool before, but have researched and developed with similar applications. I'm not sure what the performance hit of adding the database is, but they are designed for situations with a lot of events being processed quickly and maintaining a high event throughput.

At the very least, perhaps event-driven architectural models might be something you can look at and adapt as necessary to your particular problem.

Answer 2

You can store the events and subscribers in redis http://redis.io/

Since Redis is a data structure server. You can use the Hash data structure to keep track each instances of your subscriber by creating a key as the name of the subscriber and storing other important information such as ID, transport, and e.t.c in json format. Same can be done for the events also.

The benefit of using the hash is that you get to retrieve your subscriber information as needed in O(1). You could also make use of the hash structure for linking up subscribers to a particular topic