4

I have designed a simple PoC system that processes a feed of prices that tick. It consumes a stream of "Ticker, Price" objects off JMS and updates a map, so that the map simply contains the latest price for each ticker. So far so good.

Now for the final design the system needs to be scalable. The first naive approach is to use a distributed map (eg Hazelcast) and simply run multiple copies of the stream processing module in parallel. If the prices are coming in on a JMS queue, then each message will be picked up by only one of the modules, and I can simply spin up more modules to add capacity. However, there is an issue with this approach - how do I guarantee that messages will be processed in the right order?

If I receive messages like:

VOD 32.4
VOD 35.6
VOD 34.2

In a single thread, I know that they will be processed in order they are received, and my final price will be 34.2. But when running multiple modules, it could very well be that the module processing the price "34.2" completes before the one processing "32.4", and so my final price will be incorrect.

Let's say I have prices like:

VOD 32.4
IBM 42.1
VOD 35.6
IBM 45.3
VOD 34.2
IBM 44.2

How can I distribute my computation in such a way that one module will process all the "VOD" prices and the other one will process all the "IBM" prices? And then, if one of the modules gets shut down and there is only one remaining, it will need to process both "VOD" and "IBM"?

I have looked at Akka as a framework which looks very interesting - it has clustering which should give me the scalability and a "Consistent Hashing" router which I believe might do exactly this "send all messages for a particular key always to the same instance", have I understood correctly? I am also investigating using something like Apache Storm, but I have not been able to figure out how it would solve for this case.

Any suggestions much appreciated, I'm sure many people have solved this exact issue before!

  • It looks like you need guaranteed ordering per ticker, otherwise things can run in parallel in any order. Is this correct? – 9000 Apr 1 '16 at 22:20
  • Exactly. It appears that the consistent hashing router will give me this, so that all IBM messages go to one actor (in order) and all VOD messages go to another actor (also in order). It doesn't matter in this case if the second VOD message is processed before the first IBM message etc. One of the things I'm currently stuck on is can I scale a router? Or will the router become a bottleneck? Also, can I cluster the router so that if the server running the router goes down it won't take down the whole application? – Matt Apr 2 '16 at 11:34
0

For future visitors: the easiest/best solution for this (in my opinion) is to swap out JMS for Apache Kafka, using a single group id for all consumers and making the partition key of the topic the ticker. This will automagically give you a scalable number of consumers that consume a particular subset of mutually exclusive keys. Problem solved!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.