1

I have a Python script launching a C++ executable.

The C++ executable is a multithreaded program that usually takes several hours to run. The way the C++ code is written, it will run on all the cores of the CPU if possible.

However, when I run the Python script and check my task manager, I read that the Python script is only using 30% of my CPU. I think this 30% includes the CPU usage of the subprocess running the executable as well, because the executable appears in the task manager with 0% CPU usage (but it's indeed running and producing results). Sometimes the executable will spike to 7% CPU usage and go right back down to 0%.

Is there a way I can increase the CPU usage of this subprocess to save time?

EDIT: Maybe i should specify that this Python script is communicating with the C++ subprocess through a pipe. While it's running, the subprocess sends back a lot of information through the pipe that the Python script uploads to a database which might explain the high CPU usage of the script.

Improve this question
3
  • 1
    My psychic powers say processor affinity.
    – Kevin
    Commented Mar 31, 2017 at 5:57
  • 3
    It is not clear what you are asking. Does your C++ program only use 30% of the CPU power? Does the python script that launches the program use only 30%? Does the program use 100% when started directly, but only 30% when started from the python script? Something else?
    – Bart van Ingen Schenau
    Commented Mar 31, 2017 at 6:38
  • I had trouble expressing the problem clearly, but I edited the question. Let me know if it's clearer.
    – Chuque
    Commented Mar 31, 2017 at 8:41

2 Answers 2

Reset to default
1

The way the C++ code is written, it will run on all the cores of the CPU if possible

Well, did you actually test this, or do you just assume this? What happens if your C++ is started directly from the command line?

However, when I run the Python script and check my task manager, I read that the Python script is only using 30% of my CPU. I think this 30% includes the CPU usage of the subprocess running the executable as well

That is definitely wrong. If your Python script uses 30% of the CPU, it is doing the work by itself, and not the subprocess. Impossible to say what your script does without seeing the code, but my best guess is it probably wastes CPU cycles by waiting for the end of the subprocess in a very inefficient manner.

the executable appears in the task manager with 0% CPU usage (but it's indeed running and producing results

This behaviour might happen when the executable is running with a low priority, whilst your Python process has a higher priority and blocks the execution of the C++ program. Other things which could impose blocking behaviour are shared file access or other shared resource access. I would check for these kind of things.

Improve this answer
4
  • Maybe i should specify that this Python script is communicating with the C++ subprocess through a pipe. While it's running, the subprocess sends back a lot of information through the pipe that the Python script uploads to a database which might explain the high CPU isage of the script.
    – Chuque
    Commented Apr 1, 2017 at 1:22
  • @Chuque: as I already wrote, shared resource access is probably where to look for the problem. And a pipe is a shared resource. I am pretty sure one needs to dive into your code and analyse in detail what your two programs are doing exactly. Since you asked here on SE, I guess you were hoping for an "easy" way to solve your problem without thinking too much. There is none. Finding bottlenecks, especially in context of concurrent programs is hard, there is no easy way around it.
    – Doc Brown
    Commented Apr 1, 2017 at 7:10
  • Fair enough. Looking for a bottleneck is exactly what I need to do. Unfortunately I can't delete this post.
    – Chuque
    Commented Apr 1, 2017 at 7:34
  • @Chuque: well, reading your post again, it sounds the C++ program is mostly waiting for the Python script until it has written the data into the DB. Maybe you can add some kind of test mode where the script does not write anything into the database, or another test mode where the C++ program does not use the pipe? That might give you an idea which part is the bottleneck.
    – Doc Brown
    Commented Apr 1, 2017 at 7:43
1

Python threads are not good for CPU-bound load, they only work well for I/O-bound load, because of the global interpreter lock (aka GIL).

If you want to use all cores with a CPU-bound task, either use multiple Python processes, or use a different language.

Improve this answer
2
  • I'm not using Python threads here but a Python subprocess
    – Chuque
    Commented Apr 1, 2017 at 1:19
  • 1
    Did you profile your Python code? Did you plot a chart of CPU usage per Python process? Did you notice which core is each process using? Did you at least log, with timestamps, when a Python process receives a chunk of work, and when it finishes it? It's hard to tell what's happening without seeing more.
    – 9000
    Commented Apr 1, 2017 at 1:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.