I'm looking to do the following for 1000s of items:
1) get time series data for an item from file 2) calculate mean and standard deviation 3) calculate final calculation using mean and standard deviation 4) add value to list
Currently the file is one huge CSV of all items and all dates so there isn't much use multithreading that part so I will load it into memory.
I would like to do the calculations via multithreading and am hoping to use the TPL (Task Parallel Library).
I have a good idea of how to do this for example I could use a parallel for each and do the following:
1) get time series data for specific item 2) calculate mean 3) calculate standard deviation 4) calculate final calculation 5) add to thread safe dictionary
Even though this is multithreaded it still is very sequential within the thread itself so I thought of the following would be better:
Queue for timeseries data retrieval Queue for items to process for mean and standard deviation Queue for items to process for final calculation
Couple of threads per queue picking up and working on each item.
Basically will there be much benefit in me have more control or shall I just use the parallel for each?