Python: Counter for Simultaneous calls of a PHP script?

Question

I'm trying to create a queue processing script that's purpose is to find particular files and apply actions based on specific parameters, with the result being to call a PHP script that serves to process a list of message files that we generated and passed to it.

One of the things I need to specify in a config file the maximum number of concurrent calls to the PHP processor that can be running at a given time, and if max number of calls is >= then loop until <=

Also need to specify a timeout variable 'tOut' in seconds...

So basically it looks for the oldest file in the directory, locates a variable in that filename 'chCode', searches for the next oldest files in the directory with the same chCode and reprefix them and output file a list of those reprefixed paths.

Then using Popen or Pcall the script with run 'php /path/to/filepathlist' and if: it returns output '00' then continue... it returns any text other than 00 then reprefix and mv.. it returns nothing for tOut secs then reprefix and mv.

The counter is the part I'm really needing help with, as this needs to be proper accurate and high performance with batches of hundreds of thousands of outgoing text messages being sorted using this. I can confirm the counter from the PHP script but as said it needs to be accurate.

Using Python 2.7 on CentOS6.

Answer 1

If I understand your question correctly, the calls to PHP are asynchronous but you need to throttle them, right?

Use a semaphore.

http://docs.python.org/2/library/threading.html#semaphore-objects

throttle = threading.Semaphore(NUMBER_OF_CONCURRENT_EXECUTIONS)

Then when you call PHP, do it via

with throttle :
    ... call php, wait for timeout, etc ...