0

I have been working on a python project for sometime which has been structured in following manner:

We have a function like this:

def execute_cmd(cmd):
    (out, err) = subprocess.Popen(cmd).communicate()
    return (out, err)

and this function is used over and over to execute binaries of a third party library.
I did a little bit research and understood that we can connect to the third party library over sockets, and communicate by sending JSON payloads.

I am trying to understand which approach is better and why?
From what I understand, everytime we create a subprocess, underlying OS has to create a PCB for it, allocate resources and deallocate resources after completion. However, I see that it is really common practice in python programming to create subprocesses. This has left me completely confused.

Which approach is better and why?

1

On the surface, it looks like your trade-off is between performance and complexity.

Performance As you point out, there's added performance costs to executing a subprocess.

Complexity For the 3rd-party library to use a socket it must (obviously) be running. So your application will have to take on the added complexity of making sure that 3rd-party tool is running before you try to connect to the socket. Alternately, you could assume it's running, then when the socket connection fails, restart it.

Two more points on complexity:

  1. Using their socket interface will tightly couple your code to theirs while the subprocess route leaves a nice clear delineation

  2. Using sockets means you have a lot more cases/states to test and setting up tests for socket communication can be onerous.

In my opinion, the performance-hit is preferable to the added complexity. It's more easily testable and more decoupled.

  • I did not understand the decoupling. After executing subprocess, we would still need to check return code, error messages (if any), and success messages. In case of error, we need to do string comparison (or parse error msg) – Shriroop Joshi May 28 at 20:54
  • @ShriroopJoshi based on the information you've given, the subprocess route is more loosely coupled than the socket route. Maybe you will always use this 3rd-party tool and it's the only one that does what it does. In that case you're tightly coupled to it regardless of the code. If it's possible that you need to swap it out and other offerings are stand-alone executables, then you may want the flexibility of easily swapping it out OR if you plan to merge more tools then you'd also want that subprocess infrastructure in place. All of that goes into your decision, not just "is this faster"? – Matthew May 29 at 11:46
1

It depends.

Using a subprocess is probably simpler/easier to implement, but like you've said, will probably perform more poorly than reusing a single process for multiple requests (there may also be overhead of initialization of some sort in the subprocess).

The question is does that matter for your use case? If you're doing 1 request/second, the extra overhead probably doesn't matter, while if you're doing thousands, it might matter very much. To be sure, you'll need to measure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.