So I am trying to speed up my program by using concurrency and/or multi-threading and/or process parallelism. The topics are pretty complex and I am sort of new to them so I am still trying to figure out which one to use and when.

My task (rather sub-task):

  1. Get size of a UNIX directory (recursively). In fact, I will be processing multiple directories at once.

Based on what I understand, scanning directory is I/O bound process, and, as a result, decided to use threading instead of multiple processes.

Here is what I tried (functions work but the results are not really what I expect):

My dircetory scanning function - utils.py:

def get_path_size(path):
    """Returns total size of a file/directory.

        path: File/directory path.

        Total size of a path in bits.

    # Size in bytes/bits (B).
    total = 0

    if os.path.isdir(path):
        with os.scandir(path) as direc:
            for entry in direc:
                if entry.is_dir(follow_symlinks=False):
                    total += get_path_size(entry.path)
                    total += entry.stat(follow_symlinks=False).st_size
        total += os.stat(path).st_size

    return total 

Here is my multi-threaded function that calls the function above - file1.py:

import concurrent.futures

def conc(self):
    reqs = [{'path': '/path/to/disk1'}, {'path': '/path/to/disk2'}]

    with concurrent.futures.ThreadPoolExecutor(max_workers=12) as executor:
        future_to_path = {
            executor.submit(utils.get_path_size, req['path']): req for req in reqs

        for future in concurrent.futures.as_completed(future_to_path):
            path = future_to_path[future]
            size = future.result()
            print(path, size)

And here is my function using process parallelism - file2.py:

import concurrent.futures

def paral():
    with concurrent.futures.ProcessPoolExecutor(max_workers=6) as executor:
            for path, size in zip(PATHS, executor.map(get_path_size, PATHS)):
                    print(path, size)

The reason why I am having doubts is because it seems that program finishes faster (if not faster, then about the same) using ProcessPoolExecutor rather than ThreadPoolExecutor. Based on my understanding that get_path_size() is rather I/O intensive and docs saying that ThreadPoolExecutor is more suited for I/O work, I find it surprising that paral() runs faster.

My questions:

  1. Am I doing it right overall? I mean, should I be using ProcessPoolExecutor or ThreadPoolExecutor?
  2. Any other suggestions on how to make this code better/faster etc.?

Edit #1 - Test results:

I ran 5 tests for each of the 3 options (each test was ran one after another on a non-loaded machine): non-parallel, ProcessPoolExecutor, and ThreadPoolExecutor.

Total size of all directories was 65GB in this testing. Yesterday, I ran these tests on directories with total size of ~1.5TB and the results were pretty much the same, relatively.

Machine spec:

CPU(s):                20
Thread(s) per core:    1
Core(s) per socket:    10
Socket(s):             2

Non-parallel run-times:

Duration 38.25443077087402 seconds
Duration 16.98011016845703 seconds
Duration 21.282278299331665 seconds
Duration 37.90052556991577 seconds
Duration 40.511338233947754 seconds


Duration 7.311123371124268 seconds
Duration 15.097688913345337 seconds
Duration 15.133012056350708 seconds
Duration 13.949966669082642 seconds
Duration 4.563556671142578 seconds


Duration 28.408297300338745 seconds
Duration 7.303474187850952 seconds
Duration 26.91611957550049 seconds
Duration 4.6026129722595215 seconds
Duration 3.424044370651245 seconds
  • My gut instinct in this situation is that you cannot do any optimization unless you work on OS or even HW layer. Maybe try to find OS APIs that could be used to call that would return the size of the whole directory, instead of doing it yourself? This would give OS way to use it's own optimizations. – Euphoric Apr 12 '20 at 7:43
  • @DocBrown I added test results – tera_789 Apr 12 '20 at 8:17
  • @tera_789: the test results don't seem to support what you wrote in your question - for 2 of them, ProcessPoolExecutor is faster, but for 3 of them, ThreadPoolExecutor. – Doc Brown Apr 12 '20 at 9:11
  • 1
    @DocBrown yeah I see that...it fluctuates a lot...hard to make a decision thus. At this point, I am starting to think that either network or storage device's OS is playing a big role here...sometimes it is ThreadPoolExecutor, which is faster in most tests, and, sometimes it is ProcessPoolExecutor... – tera_789 Apr 12 '20 at 9:16
  • 1
    @DocBrown The thing is that CPU usage never goes really high when I run these scans (it is mostly 20-30%), sometimes even less. – tera_789 Apr 12 '20 at 9:36

First thing to understand is that threading is a form of parallelism. The differences between separate threads and separate processes are not all that important in this case.

As you write yourself, this is a heavily I/O bound process. In fact so heavily that the code running in between the I/O access will not have any measurable impact. As such, you will not expect to find a huge difference between different approaches to parallelism. But since you are asking, ThreadPoolExecutor will allocate five times as many workers as ProcessPoolExecutor. Since all of those workers just cause overhead for no actual gain (you're still limited by the I/O going over just one or two disks), ProcessPoolExecutor will be at a slight advantage. (Try reducing the number of workers, and I think, the difference will go away).

The only way to profit from parallelism in your use case would be if you can split the I/O workload across several hard disks / storage devices, where you would use one thread / process per device.

  • I reduced the number of workers to 12 (previously it was 24) but nothing really changed...I also posted test results if you wanna take a look... – tera_789 Apr 12 '20 at 8:19
  • And, the I/O workload is actually already split across different HDs and storage devices – tera_789 Apr 12 '20 at 8:23
  • How many disks? That will be much more important to know than how many CPUs. I would expect optimum results with one thread per disk (and making sure that each thread works on exactly one disk). Also your timings show a huge amount of fluctuation. I do wonder why. Is this network bound? Are other tasks running on this system during your benchmarking? – Tfry Apr 12 '20 at 8:25
  • 1
    Phew. I do think you'll have to tell quite a few more details about your setup. Obviously it involves networking, which, indeed will introduce a whole new layer to look at (and probably much more important than the details of parallelism on the controlling machine). Further: Tens of thousands of disks? Are we talking about physical disks, there? Cause that is what will matter. – Tfry Apr 12 '20 at 8:42
  • 2
    Oh, and you're really interested in per disk total usage, not per subdirectory usage, right? In that case, using df (via os.system()) will be a lot faster, than calculating directory size, recursively. – Tfry Apr 12 '20 at 9:53

Having some activity formally I/O bound doesn't imply it can't be parallelized. As a radically marginal but expressive example, consider you have to read something from tape drivers, and a tape seek is average 5 minutes. You have to read something from two different tapes, each installed into own driver (device). If you issue requests in parallel, you'll get average time approximately 5 minutes. If to issue requests one after other, result time is 10 minutes.

If I got it right, your case is for the same request set but in a single process instead of different processes. At a glance, I'd suspect that kernel I/O scheduler differentiates threads and processes, and provides some kind of I/O bandwidth limiting with a bucket per process. Another variant is that your implementation spends too much for proper transition between Python and C land. But all these are just speculations without real facts.

The problem is that performance is really hard. Folks are spending man years to tune their code and to find a tiny detail that affects all or, vice versa, to rewrite entire layers to achieve 1-2% speedup. And, after that, next change in subordinated layers (CPU, kernel, etc.) can void all these results. So, if you see difference less that, say, 30%, just select the variant you see the best for now and switch to another task :)


Just a warning: You are measuring your execution time, apparently with no other code running at the same time. But you're not on your own. You also need to consider how you affect other code running on the same hardware. If running tasks in parallel gives you halved execution time while quadrupling total network traffic, that's not a good solution because everyone else suffers.

And of course nobody cares how long it takes if you do it once. Therefore I'd try to figure out if there are ways to cache data so that the total work done on consecutive runs is less.

  • Good point but this is a dedicated hardware so I am not really worried about others doing something else on this machine. Caching data may not be required because directory sizes change frequently. – tera_789 Apr 12 '20 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.