I am working on the design stages of an application that will a) collect data from various sources (in my case that's scientific data from serial ports), keeping track of the age of the data, b) generate real-time statistics (e.g. running averages) c) display, record, and otherwise handle the data (and statistics).

I anticipate that I will be adding both data producers and consumers over time, and would like to design this application abstractly so that I will be able to trivially add functionality with a small amount of interface code. What I'm stumbling on is deciding what communication infrastructure I should use to handle the interfaces. In particular, how should I make the processed data and statistics available to multiple consumers?

Some things I've considered:

  • Writing to several named pipes (variable number). Each consumer reads from one of them.
  • Using FUSE to make a userspace filesystem where a read() returns the latest line of data even if another process has already read it.
  • Making a TCP server, and having consumers connect and request data individually.
  • Simply writing the consumers as part of the same program that aggregates the data.
  • So I would like to hear your all's advice on deciding how to interface these functions in the best way to keep them separate and allow room for extenstions.

    • What are your latency requirements? – Deer Hunter Jun 10 '14 at 3:07
    • The latency between the aggregator and the consumers is not particularly important. 200ms should be good enough. The raw data will get a timestamp as soon as it's read into the aggregator, so that latency is a bit more important. My data streams are currently at 4 samples per second. – Thomas Shaw Jun 10 '14 at 3:49

    That sounds like a job for a Publish/Subscribe architecture, AKA Reactive programming. Although the title may sound misleading, Enterprise Integration Patterns is an excellent guide in how to build such systems in the large.

    • Thanks for this. It looks like I'll be able to get a nearly-canned solution for this, e.g. RabbitMQ. Much better than implementing my own. – Thomas Shaw Jun 10 '14 at 21:56

    Your Answer

    By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

    Not the answer you're looking for? Browse other questions tagged or ask your own question.