How can I improve the speed of scanning multiple directories recursively at the same?

Question

So I am trying to speed up my program by using concurrency and/or multi-threading and/or process parallelism. The topics are pretty complex and I am sort of new to them so I am still trying to figure out which one to use and when.

My task (rather sub-task):

1. Get size of a UNIX directory (recursively). In fact, I will be processing multiple directories at once.

Based on what I understand, scanning directory is I/O bound process, and, as a result, decided to use threading instead of multiple processes.

Here is what I tried (functions work but the results are not really what I expect):

My dircetory scanning function - utils.py:

def get_path_size(path):
    """Returns total size of a file/directory.

    Args:
        path: File/directory path.

    Returns:
        Total size of a path in bits.

    """
    # Size in bytes/bits (B).
    total = 0

    if os.path.isdir(path):
        with os.scandir(path) as direc:
            for entry in direc:
                if entry.is_dir(follow_symlinks=False):
                    total += get_path_size(entry.path)
                else:
                    total += entry.stat(follow_symlinks=False).st_size
    else:
        total += os.stat(path).st_size

    return total 

Here is my multi-threaded function that calls the function above - file1.py:

import concurrent.futures

def conc(self):
    reqs = [{'path': '/path/to/disk1'}, {'path': '/path/to/disk2'}]

    with concurrent.futures.ThreadPoolExecutor(max_workers=12) as executor:
        future_to_path = {
            executor.submit(utils.get_path_size, req['path']): req for req in reqs
        }

        for future in concurrent.futures.as_completed(future_to_path):
            path = future_to_path[future]
            size = future.result()
            print(path, size)

And here is my function using process parallelism - file2.py:

import concurrent.futures

def paral():
    with concurrent.futures.ProcessPoolExecutor(max_workers=6) as executor:
            for path, size in zip(PATHS, executor.map(get_path_size, PATHS)):
                    print(path, size)

The reason why I am having doubts is because it seems that program finishes faster (if not faster, then about the same) using ProcessPoolExecutor rather than ThreadPoolExecutor. Based on my understanding that get_path_size() is rather I/O intensive and docs saying that ThreadPoolExecutor is more suited for I/O work, I find it surprising that paral() runs faster.

My questions:

1. Am I doing it right overall? I mean, should I be using ProcessPoolExecutor or ThreadPoolExecutor?
2. Any other suggestions on how to make this code better/faster etc.?

Edit #1 - Test results:

I ran 5 tests for each of the 3 options (each test was ran one after another on a non-loaded machine): non-parallel, ProcessPoolExecutor, and ThreadPoolExecutor.

Total size of all directories was 65GB in this testing. Yesterday, I ran these tests on directories with total size of ~1.5TB and the results were pretty much the same, relatively.

Machine spec:

CPU(s):                20
Thread(s) per core:    1
Core(s) per socket:    10
Socket(s):             2

Non-parallel run-times:

Duration 38.25443077087402 seconds
Duration 16.98011016845703 seconds
Duration 21.282278299331665 seconds
Duration 37.90052556991577 seconds
Duration 40.511338233947754 seconds

ProcessPoolExecutor:

Duration 7.311123371124268 seconds
Duration 15.097688913345337 seconds
Duration 15.133012056350708 seconds
Duration 13.949966669082642 seconds
Duration 4.563556671142578 seconds

ThreadPoolExecutor:

Duration 28.408297300338745 seconds
Duration 7.303474187850952 seconds
Duration 26.91611957550049 seconds
Duration 4.6026129722595215 seconds
Duration 3.424044370651245 seconds

Answer 1

First thing to understand is that threading is a form of parallelism. The differences between separate threads and separate processes are not all that important in this case.

As you write yourself, this is a heavily I/O bound process. In fact so heavily that the code running in between the I/O access will not have any measurable impact. As such, you will not expect to find a huge difference between different approaches to parallelism. But since you are asking, ThreadPoolExecutor will allocate five times as many workers as ProcessPoolExecutor. Since all of those workers just cause overhead for no actual gain (you're still limited by the I/O going over just one or two disks), ProcessPoolExecutor will be at a slight advantage. (Try reducing the number of workers, and I think, the difference will go away).

The only way to profit from parallelism in your use case would be if you can split the I/O workload across several hard disks / storage devices, where you would use one thread / process per device.

Answer 2

Having some activity formally I/O bound doesn't imply it can't be parallelized. As a radically marginal but expressive example, consider you have to read something from tape drivers, and a tape seek is average 5 minutes. You have to read something from two different tapes, each installed into own driver (device). If you issue requests in parallel, you'll get average time approximately 5 minutes. If to issue requests one after other, result time is 10 minutes.

If I got it right, your case is for the same request set but in a single process instead of different processes. At a glance, I'd suspect that kernel I/O scheduler differentiates threads and processes, and provides some kind of I/O bandwidth limiting with a bucket per process. Another variant is that your implementation spends too much for proper transition between Python and C land. But all these are just speculations without real facts.

The problem is that performance is really hard. Folks are spending man years to tune their code and to find a tiny detail that affects all or, vice versa, to rewrite entire layers to achieve 1-2% speedup. And, after that, next change in subordinated layers (CPU, kernel, etc.) can void all these results. So, if you see difference less that, say, 30%, just select the variant you see the best for now and switch to another task :)

Answer 3

Just a warning: You are measuring your execution time, apparently with no other code running at the same time. But you're not on your own. You also need to consider how you affect other code running on the same hardware. If running tasks in parallel gives you halved execution time while quadrupling total network traffic, that's not a good solution because everyone else suffers.

And of course nobody cares how long it takes if you do it once. Therefore I'd try to figure out if there are ways to cache data so that the total work done on consecutive runs is less.