Skip to main content
1 of 2
g00dds
  • 111
  • 1
  • 5

How to terminate python queue and instruct all consumer-threads to finish their tasks?

I have a multi-threaded application. There is 1 thread that produces a resource and puts it into a queue and many (not constant amount) consumer-threads that get the resources from the queue.

When there are no more resources that the first thread procudes, the first thread must terminate the queue, that is, notify all the consumer-threads to finish their tasks and don't wait for the resource any more.

How to terminate the queue?
One approach is to put a sentinel value (like None) into the queue by the first thread so that all the consumer threads after receiving the sentinel value from the queue know that the queue has been terminated and they must finish their tasks. The problem is that the producer thread needs to put as much sentinel values into the queue as many consumer threads there are. If it puts less sentinel values than the amount of the consumer threads, the left consumer threads will wait forever for an item in the queue. Like I said, the amount of consumer-threads is not constant, and for now, the producer thread doesn't know their amount. It feels like a bad and unstable approach, because the producer must know the exact amount of consumers at the specific point in time, which might be changed.

Another approach I can think about is to have a Queue-wrapper class, the only purpose of which is to add a terminated-flag. When the queue is to be terminated, the flag is set, and all further calls to get method of the queue would result in the sentinel value. The problem with this approach is that when setting terminated-flag it doesn't kick the already waiting consumer threads from waiting, so they will still be waiting forever even after setting the terminated-flag.

I can think of some other approaches like creating my own custom queue with deque and threading.Event and the terminated-flag, but I don't feel like it's a proper solution, after all people use the built-in Queue somehow.

g00dds
  • 111
  • 1
  • 5