When it comes to understanding parallel processing (I'm going to use this term as a generic term), it's best to start out with the basics. Take it slow and simple at first.
It's my understanding that threads and processes are much the same, but threads are a light-weight work system whereas processes are heavy-weight. These are the terms used to indicate how much system work must be generated in order to support these work arrangements. This is an important matter because parallel processing causes the computer some extra work. If a task spawns a great many threads, this is more efficient than if it has to spawn an equal number of processes to do the same work.
Try to understand I/O in the broadest possible sense. It's not only about user input, or I/O to mass storage devices (disk). From the CPU's perspective a read from main memory is I/O. A cache read is I/O too, even the fastest Level 1 cache. Parallel processing allows productive work to continue even if one thread (or process) is waiting on I/O, because other non-blocked threads can continue to execute.
And never get stuck thinking that you know or control the order of execution of threads you spawn! While you can force synchronization of parallel tasks, only do so if the problem logic requires it. Synchronization slows down the pace of program execution and is bad unless strictly necessary.
Most modern CPU's are complex beasts internally and will break down even Assembly language statements into micro-instructions.
All talk of multiple CPU's versus core utilizing CPU's is just hardware implementation stuff. Even a single CPU, non-core based, can often make very effective use of parallel software. It just requires OS and application support.
The crude hierarchy of processor power (in increasing order of instruction-executing power) is:
- Single CPU, no multi-core
- Multiple CPU, no multi-core
- Single CPU, multi-core
- Multiple CPU, multi-core
Why? Multi-core CPU's can route tasks around to internal cores much faster than it takes to send a task to a different CPU. Also, multi-core CPU's routinely share cache levels, more so than single-core CPU's. That permits faster data sharing too.
The crude metrics I've seen say that sending a task from one core to another, within a single CPU is approximately 10X faster than having to send that task off-chip, to another CPU.
Note too that shared-nothing architectures, like supercomputers routinely use, abstract this up even higher. Then you have multiple whole computers all working together as a coordinated unit. The instruction processing power goes up but so does the inter-process communications (IPC) overhead. This is directly analogous to the increase in power with attendant IPC overhead found between multiple single-core CPU's and one multi-core CPU. All found within a single computer with a single operating system.
Here are some general guidelines for programming parallel processing:
- implement as many parallel tasks as you can, as the logic will permit;
- stay away from trying to coordinate the number of subtasks you spawn to correlate with your current CPU design;
- stay away from second-guessing the CPU's instruction processing abilities;
- stay away from process synchronization wherever possible;
In fact it's a good design pattern to not design to your current hardware, as much as possible. Software can have a very long life while hardware has a relatively short and fixed lifespan. Therefore bet on (by spending time on) the software more so than the hardware. The hardware will change, you can count on it.