Partially parallel producer-consumer pattern with internal state

Question

I need to implement a producer-consumer pattern for reading, processing and saving electrical values. I have to implement this in C# .NET 4.6.1.

I try to describe this in great detail, so that there hopefully is no misunderstanding about the goal I want to achieve. I have an idea how to do it, but it may not be the best way.

I'd like to know what you think about my idea and how you would suggest to solve it.

The situation

The values are collected by another program from several data loggers. These values are stored in files for each data logger. The software I am writing has a configuration file, that specifies the directory of the input files, where to output the results, what format is used for the input, what format is used for the output etc. It is important that for each data logger a definition exists on how to process each (electrical) value.

The number of files, their size and the number of data loggers can vary greatly. I would assume about 1-200 data loggers, possibly more.

The Goals

I want to have parallel processing for each data logger's files with concurrent I/O. This means while the program reads new files, the content of previously read files gets processed and the results of previous calculations are written into files. The processing should happen in parallel between data loggers.

It is important that the processing of values cannot be parallelized within. Values of one data logger are processed sequentially (chronologically). To optimize throughput and limit memory usage, there has to be buffering between I/O and processing, possibly also between processing steps.

A little sketch

+----------+               +----------------+               +-----------+ 
|Read files|---->Queue---->|Process values  |---->Queue---->|Write files|
+----------+         |     |of data logger 1|               +-----------+ 
   (Task)            |     +----------------+                  (Task) 
                     |     +----------------+
                     |---->|Process values  |
                     |     |of data logger 2|
                     |     +----------------+
                     ...       (n Tasks) 

The Idea

My idea is to use the Task Parallel Library (TPL) Dataflow. Which could look like this.

+--------------+     +--------------+     +----------------+               +-----------+ 
|TransformBlock|     |  BufferBlock |     | TransformBlock |               |ActionBlock|  
|  Read files  |---->|     Queue    |---->| Process values |---->Queue---->|Write files|
+--------------+     +--------------+     |of data logger 1|               +-----------+ 
   (1 Block)                        |     +----------------+                 (1 Block) 
                                    |     +----------------+
                 conditional linking|     | TransformBlock |
                                    |---->| Process values |
                                    |     |of data logger 2|
                                    |     +----------------+
                                    ...       (n Blocks) 

As the data being read is not meant for every TransformBlock I would use conditional linking. Each TransformBlock that processes values would be generated in a loop depending on how many data loggers we have, together with an instance of the class containing the processing functions.

It is important that there are several objects for processing, because the calculations have a state. A simplified example for a state could be the last calculated value, if I'd like to sum all incoming values. Another could be the last timestamp, so I can validate that the new value is chronologically correct. The state would of course exist per object, thus per block and data logger. No locks should be necessary.

The maxDegreeOfParallelism would always be set to 1, because the processing within the TransformBlocks has to be sequential.

The conditional linking might impact performance, because it always has to check where the file belongs to. If there would be a way to directly choose a block to send a message to, this could save some time. I choose not to use a block for each data logger to read the files, because multiple Tasks for I/O might hinder performance.

What do you think about my idea? How could it be improved? Would you take an entirely different approach?

I need to be honest, I have already posted a very similar question on Stack Overflow, but it seems that I did not explain my problem very well.

Answer 1

I have thought this through and a consumer-producer pattern with Tasks and BlockingCollections should be the easiest and most efficient solution.

+----------+                          +----------------+                           +-----------+ 
|   Task   |                          |      Task      |                           |   Task    |
|Read files|-----BlockingCollection-->|Process values  |----BlockingCollection---->|Write files|
+----------+   |                      |of data logger 1|  |                        +-----------+ 
               |                      +----------------+  |             
               |                      +----------------+  |
               |                      |      Task      |  |
               |-BlockingCollection-->|Process values  |--|
               |                      |of data logger 2|  |
               |                      +----------------+  |
                ...                       (n Tasks)        ...

The BlockingCollections get created first and are passed into the constructor of the Task reading files. The BlockingCollection needed for a data processing Task is also passed to the object representing the processing unit for a data logger. Each Task for data processing runs a function of the corresponding object, which also holds the internal state.

The Task reading the files is able to distinguish between the different data logger files and can pass the data to the correct BlockingCollection. Depending on the strategy used, it already knows where to pass the data opposed to the TPL Dataflow with conditional linking.