0

I am trying to design and implement a solution for the following problem I am facing in one of my projects.

There are n (say 30) clients that send me data points of the form {timestamp, object}, where ‘timestamp’ comes in strictly ascending order.

I (server) need to listen for them and process the aggregated data from all clients for each time stamp in the following cases:

  1. I hear from all 30 clients for a single timestamp.
  2. I time out waiting to hear from a subset of the clients. This time out starts after I receive data for a time stamp for the first time.
  3. A subset of clients send data of timestamp higher than the one I need to process for (in this case I should process before the time out for whatever data I aggregated).

Is there a better way than maintaining bitmap of 30 clients for each timestamp, using background threads to continuously check the bitmaps and receive messages from forked listener process if higher timestamp is seen for any client? I would like something that is fast because the amount of data to be received in one hour is around 200 GB.

Any suggestions are appreciated, I don't have much experience in C++ systems programming. Please comment if you need any more details about anything, I apologise for missing anything.

Edit: I read about ZeroMQ, Kafka and RabbitMQ. Is MQ a good model for this kind of a problem?, especially with the amount of data I need to process?

Improve this question
4
  • 1
    How are the clients able to send a package with the same timestamp? Do all 30 of them need to submit it in the exact same millisecond?
    – BlueWizard
    Jul 30, 2017 at 3:03
  • @BlueWizard : The clients would send different data for each time stamp, how are they able to do that is not relevant/known to me. They may submit it in the same millisecond or over a period, hence the need for time out.
    – zorro
    Jul 30, 2017 at 3:27
  • How frequently does one client send a data point and how wide is the time window in which all clients should have sent the data point for timestamp T?
    – Bart van Ingen Schenau
    Jul 30, 2017 at 6:41
  • @BartvanIngenSchenau A client sends a data point every 1 milli second. The time out is 60 seconds.
    – zorro
    Jul 30, 2017 at 7:27

1 Answer 1

Reset to default
1

I would do some reading on distributed stream processing. There are tools that can operate on distributed streams (think Apache Kafka) that can do similar things to your requirements. Apache Spark Streaming and Apache Trident are two examples of such tools.

Specifically they allow you to write stream processing applications that do a lot of heavy lifting for you. For example, you can set up a sliding window that will give you records within those windows of time. You should be able to implement your requirements with windowed functions.

Since you're performing these operations as data enters your system, you cando this processing in real time, and it's much more scalable than searching a 200GB data store.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.