0

I am trying to design and implement a solution for the following problem I am facing in one of my projects.

There are n (say 30) clients that send me data points of the form {timestamp, object}, where ‘timestamp’ comes in strictly ascending order.

I (server) need to listen for them and process the aggregated data from all clients for each time stamp in the following cases:

  1. I hear from all 30 clients for a single timestamp.
  2. I time out waiting to hear from a subset of the clients. This time out starts after I receive data for a time stamp for the first time.
  3. A subset of clients send data of timestamp higher than the one I need to process for (in this case I should process before the time out for whatever data I aggregated).

Is there a better way than maintaining bitmap of 30 clients for each timestamp, using background threads to continuously check the bitmaps and receive messages from forked listener process if higher timestamp is seen for any client? I would like something that is fast because the amount of data to be received in one hour is around 200 GB.

Any suggestions are appreciated, I don't have much experience in C++ systems programming. Please comment if you need any more details about anything, I apologise for missing anything.

Edit: I read about ZeroMQ, Kafka and RabbitMQ. Is MQ a good model for this kind of a problem?, especially with the amount of data I need to process?

  • 1
    How are the clients able to send a package with the same timestamp? Do all 30 of them need to submit it in the exact same millisecond? – BlueWizard Jul 30 '17 at 3:03
  • @BlueWizard : The clients would send different data for each time stamp, how are they able to do that is not relevant/known to me. They may submit it in the same millisecond or over a period, hence the need for time out. – zorro Jul 30 '17 at 3:27
  • How frequently does one client send a data point and how wide is the time window in which all clients should have sent the data point for timestamp T? – Bart van Ingen Schenau Jul 30 '17 at 6:41
  • @BartvanIngenSchenau A client sends a data point every 1 milli second. The time out is 60 seconds. – zorro Jul 30 '17 at 7:27
1

I would do some reading on distributed stream processing. There are tools that can operate on distributed streams (think Apache Kafka) that can do similar things to your requirements. Apache Spark Streaming and Apache Trident are two examples of such tools.

Specifically they allow you to write stream processing applications that do a lot of heavy lifting for you. For example, you can set up a sliding window that will give you records within those windows of time. You should be able to implement your requirements with windowed functions.

Since you're performing these operations as data enters your system, you cando this processing in real time, and it's much more scalable than searching a 200GB data store.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.