System design for a client server problem

Question

I am trying to design and implement a solution for the following problem I am facing in one of my projects.

There are n (say 30) clients that send me data points of the form {timestamp, object}, where ‘timestamp’ comes in strictly ascending order.

I (server) need to listen for them and process the aggregated data from all clients for each time stamp in the following cases:

1. I hear from all 30 clients for a single timestamp.
2. I time out waiting to hear from a subset of the clients. This time out starts after I receive data for a time stamp for the first time.
3. A subset of clients send data of timestamp higher than the one I need to process for (in this case I should process before the time out for whatever data I aggregated).

Is there a better way than maintaining bitmap of 30 clients for each timestamp, using background threads to continuously check the bitmaps and receive messages from forked listener process if higher timestamp is seen for any client? I would like something that is fast because the amount of data to be received in one hour is around 200 GB.

Any suggestions are appreciated, I don't have much experience in C++ systems programming. Please comment if you need any more details about anything, I apologise for missing anything.

Edit: I read about ZeroMQ, Kafka and RabbitMQ. Is MQ a good model for this kind of a problem?, especially with the amount of data I need to process?

Answer 1

I would do some reading on distributed stream processing. There are tools that can operate on distributed streams (think Apache Kafka) that can do similar things to your requirements. Apache Spark Streaming and Apache Trident are two examples of such tools.

Specifically they allow you to write stream processing applications that do a lot of heavy lifting for you. For example, you can set up a sliding window that will give you records within those windows of time. You should be able to implement your requirements with windowed functions.

Since you're performing these operations as data enters your system, you cando this processing in real time, and it's much more scalable than searching a 200GB data store.