Due to the Global Interpreter Lock, multithreading in Python does not affect parallelism. This limitation is avoided in the multiprocessing library by spawning new processes instead. But threading does improve responsiveness in certain (IO-bound) situations. (from this answer on SO).

My situation is as follows. A Python process is communicated with using stdin and stdout. One line corresponds to one message and is met with a result line. Currently I have a single process consuming the messages one by one and producing a result. However, the processing involves moderately heavy calculations, so better performance could be achieved with parallel computations of different requests.

After reading up on the differences and uses of threading and multiprocessing, I've come up with a structure that would hopefully suit this task.

  • Messages ought to be handled asynchronously to allow for reading stdin and distributing work before returning results. Two threads could be spawned to handle the reading and writing.
  • Multiple processes could be spawned to respond to those requests. An example from the multiprocessing documentation: here (#3: using Queues)

As the example higlights (with random sleep times), the response order is not fixed. Jobs are returned in the order they finish.

Is this a sensible design? Furthermore, what could be done to preserve the order of messages in the pipeline?

  • A sensible design is one that adequately meets your software requirements. Does this design adequately meet your software requirements? Aug 1, 2018 at 15:34
  • As you noted, due to the GIL there's no multithreading in Python. you can use multiprocessing but you should note that the cost of creating a new process is very non-negligible. So before you spawn a new process I would consider just using asyncio and see where that gets you. The time spent waiting for IO is so enormous that just switching to asyncio will give you a pretty enormous speedup without needing to start multiple processes. It also simplifies design since you don't need mutexes of any sort since you're only running one process that just jumps around a lot. Aug 1, 2018 at 15:45
  • @RobertHarvey Yes. Yes it does. Yet, there are two fine suggestions below that improve my sensible design. Let's say instead: "optimal", or even better let's rephrase the question: "is there a way to improve this design?" .~
    – Felix
    Aug 1, 2018 at 16:38
  • @NickChapman Yeah, thanks for the suggestion. I'll try that. But the IO is somewhat neglible compared to the time it takes to process the information received. Well, don't know about neglible but I imagine it's not the bottleneck. I'll investigate.
    – Felix
    Aug 1, 2018 at 16:40

2 Answers 2


Yes. I've implemented this a couple times successfully.

What I did was:

  • Create a utility service python application that reads a JSON message, does a computation and prints an answer. (this is slow but easy to debug/test).
  • Implement a slight enhancement on that process, so that the service (called) python script keeps reading from stdin and processing records until it gets to EOF (saving all the startup time of the service) - this makes things much faster.
  • Then implement an OPTIONAL argument to the service --binary - so that it can be run with either binarized inputs (if called with --binary) or by default expecting text (JSON) inputs.
  • Finally, in the calling layer, you can spawn as many of these back end computation processes as you want (I did one per system core to run the device CPU hot).

With all this - together, you have a very simple, testable (myPythonApp.py < test.json) architecture, that performs about as well as python can (so not great ;-)). But - its ALSO an architecture where you can CALL this from whatever language system you want, and if you need the whole thing to run faster, you CAN rewrite the backend python service in C++ (and easily test it gets same results as existing python one but faster).

  • Oh I see, you wouldn't implement the multiprocessing in the service itself, just use multiple services. Did I understand you correctly?
    – Felix
    Aug 1, 2018 at 14:35
  • Correct. One process for each 'compute engine'. That compute engine can be written in any language (eg. python) and do computes and return results. Caller can be written (in any language but your case python) - and all it needs to do is spawn N processes, connected by pipes, and read from them with select. THIS part I've never written (exactly) in python, but have written in C++ (though I did something similar in python reading from sockets for overlapped network activity). So when you run 'top' - you should see 1 python compute engine for each core at 100%CPU and the controller using little. Aug 1, 2018 at 15:36

Since the responses have to be in the same order as the requests, using multiple processes will not speed up any individual response – you still have to wait until all previous requests have been completed. But you can now process multiple requests at once. Then:

  • the waiting time until work on a request is started might be reduced
  • the throughput of the system is increased

Do not keep a queue of requests that are consumed by a pool of workers. Instead, keep a queue of pending workers. You can then wait until the first worker in line has produced a response. This makes it easier to keep track of the order.

Instead of managing worker pools and queues yourself, consider whether you can use Python 3.6+ asyncio. This lets you easily use a process pool instead of a thread pool, and abstracts over many details. Also, you will be able to use futures to represent pending worker results. You then simply await the next future, in the order that you created these futures.

  • That's what I ment with parallel processing of different request. My bad if it wasn't clear. But that's a good suggestion, queueing (English plz) the workers instead of the jobs. But I wasn't going to thread pool the workers, but use multiprocessing, which seems to do a similar thing based on your answer. Those 'futures' seem interesting as well.. I'll have to try things out.
    – Felix
    Aug 1, 2018 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.