I am trying to write a script that will take multiple paths to files on various servers, search them all simultaniously, and return a single list of results to a user. Initially, I was just using Python threads to do this, however I quickly ran into some noted problems:

1. I wasn't controlling how many threads could be kicked off. So if someone sent 100 files to query for one server, you'd have 100 threads kicked off on that machine, which was bad news.

2. The results I was getting back were incomplete and varried drastically. If I ran the searches linearly (no threads), I would get complete results, but it would take a long time. I concluded based on this and some personal research that I was not taking a thread-safe approach and began looking into the Queue module.

I've ended up with something like this...

def worker():
   while q.qsize != 0:
      cmd = q.get()
      # kick off a bash command that zgreps files from different servers
      p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
      results.extend(''.join(p.stdout.readlines()).split('\n')[:-1])
      q.task_done()

NUM_WORKER_THREADS = 10
results = []

q = Queue.Queue()
for i in range(NUM_WORKER_THREADS):
   t = threading.Thread(target=worker)
   t.daemon = True
   t.start()

""" Code to generate input commands needed here """

for c in commands:
   q.put(c)

q.join()

""" Post processing of collected *results* array"""

After putting some thread-pool constraints around my program, and also checking in each thread if there is anything still in the queue, my results are inline with what I would expect. After testing, the results match the output of a single threaded approach (except it is much faster).

My questions are as follows:

1. Is my approach thread-safe? Is there any chance one of the 10 worker threads could overwrite anothers attempt to extend the results array? I am worried that I have just decreased the chance of overwriting to take place, by allocating a smaller thread pool to handle the input, but have not actually solved the problem.

2. I understand from reading that queues are supposed to be thread-safe. However, when I eliminate thread pools and don't check for the queue size in my threads, I can reproduce the same problem I was having before with large volumes of input. Can someone explain why that is?

Thanks in advance!

I am trying to write a script that will take multiple paths to files on various servers, search them all simultaneously, and return a single list of results to a user. Initially, I was just using Python threads to do this, however I quickly ran into some noted problems:

1. I wasn't controlling how many threads could be kicked off. So if someone sent 100 files to query for one server, you'd have 100 threads kicked off on that machine, which was bad news.

2. The results I was getting back were incomplete and varied drastically. If I ran the searches linearly (no threads), I would get complete results, but it would take a long time. I concluded based on this and some personal research that I was not taking a thread-safe approach and began looking into the Queue module.

I've ended up with something like this...

def worker():
   while q.qsize != 0:
      cmd = q.get()
      # kick off a bash command that zgreps files from different servers
      p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
      results.extend(''.join(p.stdout.readlines()).split('\n')[:-1])
      q.task_done()

NUM_WORKER_THREADS = 10
results = []

q = Queue.Queue()
for i in range(NUM_WORKER_THREADS):
   t = threading.Thread(target=worker)
   t.daemon = True
   t.start()

""" Code to generate input commands needed here """

for c in commands:
   q.put(c)

q.join()

""" Post processing of collected *results* array"""

After putting some thread-pool constraints around my program, and also checking in each thread if there is anything still in the queue, my results are inline with what I would expect. After testing, the results match the output of a single threaded approach (except it is much faster).

My questions are as follows:

1. Is my approach thread-safe? Is there any chance one of the 10 worker threads could overwrite anothers attempt to extend the results array? I am worried that I have just decreased the chance of overwriting to take place, by allocating a smaller thread pool to handle the input, but have not actually solved the problem.

2. I understand from reading that queues are supposed to be thread-safe. However, when I eliminate thread pools and don't check for the queue size in my threads, I can reproduce the same problem I was having before with large volumes of input. Can someone explain why that is?