Can multi-threading improve performance of an IO-bound process?

Question

I am trying to understand the difference between CPU Bound vs IO Bound process. ChatGPT suggested that multi-threading/parallel processing can help a CPU bound process; However, I think that having multithreading can help with IO Bound process.

Let's say the application is talking with some API, and has to make various API calls. If we have multi-threading, we can make those API calls in parallel, and boost the performance.

Answer 1

I/O performance has two aspects that you must distinguish: latency and throughput. One measures how long you have to wait for the first byte of a response, the other how long you have to wait additionally for the last byte.

As pjc50 said, if your network link is already saturated (no more packets can go through) it doesn't matter how smartly you organize your program - even a quantum computer couldn't help you.

But if your bottleneck is mainly lag (e.g. a web API that takes five seconds to respond at all), then sending requests in parallel will absolutely speed things up, and multi-threading is a convenient way of doing that.

Answer 2

If you are truly I/O bound, such as your network link being saturated, then no matter how many processes you run you will not be able to transmit requests and receive responses any faster.

If you are I/O bound, adding processes does not give you more I/O capability.

Answer 3

Let's say the application is talking with some API, and has to make various API calls. If we have multi-threading, we can make those API calls in parallel, and boost the performance.

This is correct. ChatGPT's answer is the way it is because there are ways to achieve this goal that better make use of the system resources. It doesn't understand anything which results in this superficial answer.

Assume, for the sake of argument, that you have an application which is non-threaded and makes heavyweight IO calls such as retrieving data from a webservice. In this situation, your application is paused or 'blocked' while that IO is happening. A common solution to this in years past was to add multithreading. That allows the blocked thread to wait on IO (it wasn't doing anything anyway) while other threads continue to execute. This works even on a system where there's only one CPU because the OS can switch thread contexts without any direct interaction from the application.

This is great but we can do better. Threads aren't free. They take up system resources and switching thread contexts also has a real, measurable, cost. There are limits to the number of threads a given system can run without becoming so bogged down in thread management that it can't do much else.

This brings us to 'non-blocking IO'. In this approach, the thread no longer stops executing while waiting for IO. Instead, it 'drops off a message at the IO mailbox' does some other things and comes back later to get the response. This allows the application to avoid IO waits without additional threads and the overhead associated with them. You can do everything on one thread. It's hard to understate the value of this. The addition of async/await to Python and made it a much more appealing language for IO-heavy solutions.

However, non-blocking IO only helps because when you are waiting on IO, there's nothing to do on the CPU. If you want to execute computations at the same time within in a single application, non-blocking IO doesn't help. And really, running multiple threads on a single CPU will not help either: the CPU only works on one thread at a time. To truly have parallel/concurrent processing within a single process, you need multiple threads running on more than one CPU.

This brings us back to the original claim: multithreading can help with CPU-bound processes. That might be true, depending on what kind of processing it is. Multithreading can also help with IO-bound processes but it's not really the preferred approach. You are not wrong, but an even deeper understanding can allow you to build more optimal solutions.

Answer 4

In general, multi-threading access to a single resource (CPU, network, disk, etc.) will not improve performance. Multi-threading will improve performance when the theads access different resources.

Multiple cores improve performance by increasing the capability of the CPU resource.

Answer 5

From the CPU point of view, I/O is most of the time waiting for something to happen elsewhere (read also "Must-know numbers for every software engineer"):

The CPU could gain time by doing something else while waiting. This does not require multithreading, just hardware interrupts and a clever software design. But the latter part is very difficult to design and even more to implement.

This is where multithreading comes in. Multithreading facilitates the design of such clever software, leaving different threads for different I/O task waiting on ice while the CPU is running other active threads. This was already true 40 years ago without hardware thread support (although multithreading at that time decreased overall system performance except for I/O):

Nowadays, multi-threaded-multi-core CPUs bring it to the next level. If you benchmark non-I/O threads on modern CPUs you will find out that within the hardware boundaries: the more threads, the (slightly) slower each separate thread will run, but cumulated performance of all the threads will be far superior than running the same computations on a single thread. So, multithreading boosts the overall system throughput by taking advantage of the hardware throughput improvements and adding the I/O waiting time optimization.

Answer 6

Multithreading is the alternative to single threading. If you’ve ever saved and watched your mouse pointer turn into a spinning thingy you’ve enjoyed the benefits of multithreading. Because while one thread is waiting for the write to complete the other is keeping you entertained.

Now sure, unless you’re doing some on the fly compression that’s not making your file write go faster but that reassuring spiny thingy sure makes it feel faster.

This assumes what’s being saved needs no extra work done to save. Which is typical.

Back in the day saving used to mean the computer fully stopped responding. Now you can watch cat videos while saving your massive spreadsheet. It’s not really faster. Just feels that way.

Answer 7

Whether I/O or CPU bound, multi-threading may help performance, under two conditions:

1. The task can be broken up into elements which can be run in parallel, instead of running all steps sequentially.
2. The thing doing the processing is capable of doing those elements in parallel without becoming overloaded.

In your question, the latter is the network (or whatever is on the other end of the I/O). If it has the capacity to accept and process many requests at once, multi-threading may help performance. If it's saturated by a single request, then issuing many requests won't help.

This is why performance work is often about experimenting and tuning... doing 2 things at once might be faster than doing 1 thing at a time, but doing 20 things at the same time might be slower.

Answer 8

Is there anything wrong with my understanding?

As long as the API is non-blocking, a single threaded system can also make those API calls in parallel. By the assumption, we are IO bound, so there is plenty of CPU time to send each request and service each response, as they arrive.

Answer 9

A trivial case is a computer with two hard drives. You can have one thread reading from the first hard drive, and another writing to the second hard drive. The most extreme example I have seen was a home-made floppy disk duplication machine which wrote to eight floppy disk drives in parallel.

Often you can change an algorithm to reduce I/O. If you have a newer Mac, any data written to a swap file (I/O bound) will be compressed before writing and decompressed after reading. This reduces the size of I/O wear and tear on the disk drive, and the compression is so fast that the operation is overall faster. More because compression makes more RAM available so swapping is not even needed and the I/O completely gone.

Answer 10

Yes, multithreading can absolutely help with I/O-bound process performance (as well as with CPU-bound performance of course).

Here is an example for I/O-bound speedup of 100%. Imagine you implement an algorithm to copy a file in a file system (plain old hard drive or SSD, mounted locally). The core loop will be something like this:

• Initialize a buffer of 1 MB
• Read up to 1 MB of data from the source file to the buffer
• Write the buffer to the target file
• Repeat until end-of-file

Assume that the source and destination files are on different physical drives, and that all other components in the computer (CPU, RAM, Caches, etc.) are much faster than the physical storage so can be neglected here.

Written without multithreading, this will take as long as it takes to read all bytes from the source file, plus the time it takes to write them to the destination file.

A multithreaded variant using a ring buffer would be:

• Tread 0:

  • Initialize a ring buffer of 10 MB
  • Initialize global multithreading-safe variables HEAD := 0, TAIL := 0, FINISHED := 0
  • Start Thread 1
  • Start Thread 2
  • Wait until both are finished
  • Exit the process

• Thread 1:

  • Start loop
    • Read up to 1 MB from the source file and place them at location HEAD * 1 MB in the ring buffer.
    • If (HEAD + 1) MOD 10 == TAIL, wait until they are not.
    • Update HEAD := (HEAD + 1) MOD 10
  • Repeat until end-of-file.
  • Set global variable FINISHED := TRUE.
  • End thread.

• Thread 2:

  • Start loop
    • Wait while HEAD == TAIL AND NOT FINISHED.
    • End thread if HEAD == TAIL.
    • Write the 1-MB block starting at TAIL * 1 MB to the destination file.
    • Update TAIL := (TAIL + 1) MOD 10
  • Repeat

The overall time for this will be variable, but roughly on the order of the maximum of the individual times for reading the whole file and writing the whole file (so half of the time used in the single-threaded algorithm, assuming those operations take roughly the same time), plus the time for writing a random small number of blocks (since thread 2 must by definition trail a bit after thread 1).

In other words, for huge files, the multithreaded approach will be roughly twice as fast (assuming reading and writing take the same time, and assuming it's separate physical devices for the two files).

The same principle works for all kinds of I/O, not only file system but also network or direct access to chipset I/O pins or or keyboard input or whatever you have.

Also note that there are very different ways to implement parallelism for I/O, not only multi-threading; for example there is the concept of multi-processing which feels somewhat similar to multi-threading but "looks" quite different. There is event-driven I/O (or multiplexing) which is implemented totally differently from the application point of view (and does not even require or use multi-threading), but will lead to similar results if the CPU required for the I/O operations is negligible.

In fact, when looking at low(ish) level Unix-style I/O sockets, it is totally normal to use these kinds of operations (i.e. the select multiplexing scheme); blocking on I/O by default is kind of a convenience programmers use when very little parallelism or scaling is required.