-1

Here my scenario: I have a client application that needs a dedicated powerful computation system that every seconds process some data in input (few kilobyte) and return a few megabytes of processed data that the client displays in a certain way. The algorithm to process the data has a sort of "precision" parameter so we can, on a normal computer, process one second of data in one second with "low resolution". Although if we want "high resolution", it could take minutes.

So that's why we need to distribute the load and we are looking into Apache Storm right now (even though Cuda or other systems are under evaluation so suggestions are appreciated but I'd like to get help for Storm here).

I've been looking into Storm in general and some tests in both Aws and Azure but what I can't understand is how to have the source and the destination to be the same sprout and so our client application. So for example I could send the data to a sprout, the sprout partitions the data to many instances of a bolt, the processed data get saved in memory to Redis, the client accesses (or call an app that accesses) the Redis db.

It seems to me the last 2 steps could be cut out and send back the processed/output data on the same channel the client sent the input data (meaning the sprout..)

Ps: once data is sent back to the client, we don't need it anymore on the "server"/storm system and no point on be available to be requested in the future.

1 Answer 1

0

I think I found a possible answer with DRPC

http://storm.apache.org/releases/current/Distributed-RPC.html

but other options are appreciated too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.