1

I'm looking to do the following for 1000s of items:

1) get time series data for an item from file 2) calculate mean and standard deviation 3) calculate final calculation using mean and standard deviation 4) add value to list

Currently the file is one huge CSV of all items and all dates so there isn't much use multithreading that part so I will load it into memory.

I would like to do the calculations via multithreading and am hoping to use the TPL (Task Parallel Library).

I have a good idea of how to do this for example I could use a parallel for each and do the following:

1) get time series data for specific item 2) calculate mean 3) calculate standard deviation 4) calculate final calculation 5) add to thread safe dictionary

Even though this is multithreaded it still is very sequential within the thread itself so I thought of the following would be better:

Queue for timeseries data retrieval Queue for items to process for mean and standard deviation Queue for items to process for final calculation

Couple of threads per queue picking up and working on each item.

Basically will there be much benefit in me have more control or shall I just use the parallel for each?

Improve this question
3
  • Depending on the size of the timed series, you may have better performance doing everything in a serial way. TPL has some overheads, and you can test the performance impact by using it simply using the Parallel.Foreach. Remember, sometimes a YEAR of records for processing is just a couple thousand items, and computers are very fast doing millions of calculations linearly. Unless you're doing a big number crunching, doing things in parallel may not be beneficial at all.
    – Machado
    Jul 28, 2017 at 13:46
  • TL;DR: Before using parallel, check: The # of records you're going to process and the complexity of the calculation you need to do with the data. Standard Deviation and Means are not really computer intensive.
    – Machado
    Jul 28, 2017 at 13:47
  • Thanks @Machado, that makes sense, I'll stick with the parallel for each and tune it if imitant fast enough
    – Simon Nicholls
    Jul 28, 2017 at 14:53

1 Answer 1

Reset to default
2

Even though this is multithreaded it still is very sequential within the thread itself

There is nothing wrong with that. You don't get bonus points for launching more threads; in fact, those come at a significant performance cost as you try to establish threading contexts and attempt to access resources protected by concurrency mechanisms. You want to have as little multithreading as possible while having enough threads to use all of your cores.

I would go with Parallel.ForEach. Don't overthink it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.