I am running 10 instances of the same executable where each executable is accessing a different 1/10 chunk of the total data that needs to be processed on a Windows Server 2012 R2. (Intel(R) Xeon(R) 2.4GHz (2 processors), 64GB RAM)
My question is if the CPU hits 100% usage, are the 10 instances still being parallel processed? Or Does the CPU start sequencing them instead of parallel processing them? Am I better off simply reducing the number of instances till the CPU usage is less than 100%?
My question is if the CPU hits 100% usage, are the 10 instances still being parallel processed?
Yes.
Or Does the CPU start sequencing them instead of parallel processing them?
Your processor probably doesn't have 10 CPU cores, so yes, it is doing swapping of them in some sequence, too. But it does not suddenly run one thread to completion when it gets to 100%. It is using swapping to give each thread a time slice; and it is doing this whether it is at 100% or not.
Am I better off simply reducing the number of instances till the CPU usage is less than 100%?
No, you want 100% CPU usage, or else you're wasting cycles!
However, that is not even close to the whole story, and, 100% CPU is not a great measurement of the efficiency of the CPU cycles that are being used.
You should probably run as many threads as your processor has CPU cores. (If you are using single threaded processes, then count those as threads.) When you run more than that, then the operating system swaps the threads into and out of the CPU cores in a balance of fairness and other policies that gives each thread a time slice here and there.
Swapping causes various kinds of overheads: direct overhead as cycles are wasted doing swapping instead of your work, and indirect overhead, which has to do with the operation of the caches — the highest speed memories that are directly on the processor die. Caches form one level of the memory hierarchy, which also includes main memory (your 16+ GB of RAM) and ultimately also includes the disc as a form of memory. Using the memory hierarchy well is the key to performance throughput.
As each thread runs for a while, it will take over part of the cache for it's own purposes. In some sense, the cache is warming up to that thread, and this is good.
When the operating system can, it will give consecutive time slice to the same thread on the same processor, and the that thread will have a warm cache to start.
Too many threads at once will cause undo cache pollution as due to their being swapped in: when a thread is first swapped in, the cache will be cold for it, and it will have to warm up before you're getting to best performance.
Yet still, processing a large data set efficiently requires not just keeping the cores busy but also broadly getting better use of the memory hierarchy.
One approach to finding a good way to slice the work, which you can tweak and refine by timing, is something like this:
Divide the whole work into chunk size that are about the size of the large processor cache (L3, maybe).
Then divide each chunk of work into sections of as many processor cores as the processor has (e.g. 2, 4), and run those. Then as those sections complete, run subsequent sections of the next chunk.
A lot has to do with the workload, and the cache architecture. Smaller L1 caches are likely to be replicated and each dedicated to a particular CPU core; larger L2 and L3 caches may be shared by all cores, though potentially with restriction such that one CPU core doesn't starve the others.
It goes to how your specific workload runs on the specific cache architecture on the machine in question. So, subdividing the chunks by the L3 size may work well, but chunks of the L2 size may work better.
One application/workload may access its section over and over in a more or less random way, while a different application/workload may only run serially thru the section start to finish. These different workloads have different efficiency aspects in the memory hierarchy, so breaking them up in some different way might help or hurt one or the other workload.
There are also better measurements you might want to take using various tooling.
What a weird question.
"Parallel processing" does not imply that threads execute in any particular order, or at any particular rate of completion, or in lockstep with regards to their individual steps, or anything like that at all. There is no guarantee as to order of beginning or ending of execution.
If you need to synchronize your threads so that they finish at the same time, the best way to do that it to write synchronization logic in your code that waits for all threads to finish before moving on to the next task. There are a variety of ways to do this depending on your programming language.
Here's a more simple-minded answer. Ignoring some power management stuff (which mostly won't be relevant in this situation anyway), a CPU core is either going full out or it's idling. So why isn't the CPU usage always at 100%? Aren't you always doing something? Why would a CPU core be idling? A CPU core idles when there is no work to do. There are two reasons there could be no work to do.
First, there could literally be nothing that needs to be done. For example, if you run a single instance of a single-threaded program and you have multiple cores, say 2, then the core that isn't running the program has nothing to do. In this case, the overall CPU usage can only be 50% at best. This is why you typically want to have at least as many threads as cores.
Second, and more common, the CPU is waiting for something. If you want to read in a file and process it, the CPU has to wait until the file is read in before it can move on to processing. Nowadays reading a file mostly doesn't involve the CPU but hard drives (including SSDs) can be many orders of magnitude slower than the CPU (depending on access patterns). In the meantime, the CPU looks for other things to do. If it can find something else to do, it switches to that, otherwise it idles.
So, a 100% CPU usage regime, means whenever the CPU looks for new work, it finds it. Anything less than 100% CPU (assuming you have at least as many threads as cores) means the CPU is often finding itself in the situation that all threads are blocked waiting for something. All other things equal, anything less than 100% CPU usage means you are wasting cycles. In practice, the bottleneck is often I/O bandwidth or memory. We talk of CPU-bound, I/O-bound, and memory-bound processes. If one of these other resources are the bottleneck, then forking more threads/processes to try to get 100% CPU usage is either not going to succeed or it's not going to make the work complete any sooner. In fact, it's likely to be harmful. If I fork two memory-bound processes, then when I switch between them, I'm going to have to swap out the memory of one to swap in the memory of the other, at which point the CPU idles waiting for the memory to swap in. High performance applications typically carefully manage scheduling to get the best disk access patterns and make sure everything relevant always fits in memory (and caches).
