writing a controller file in Python

Question

I need some advice on my idea to write a controller file in Python, and not like the C in MVC type controller but a more simpler idea. Just a Python script that controls the operation of some other Python scripts.

I will outline the idea, and any advice or ideas would be greatly appreciated.

I have to write a system that does some background tasks for a web application. The web app is in PHP and separate from the Python system I am writing, the Python will do some automated tasks and calculate and save some analytical data for the web app to use and display.

I have experience with basic Python but I have not pushed forward to learn much about class/objective Python. So far what I have written for the system is a bunch of separate py scripts that do the jobs they have to do, written totally procedurally. They all work fine, but it's not really a system yet. My initial krux was that I had to keep them all alive and running infinitely in the background. After some research I decided that the best way to do this was to have a process manager like Supervisord look after each script, I would start up the scripts and allow them to run for a certain time with a finite loop (in each script) and then they would exit, then the supervisor would restart them every time they exit and keep them alive. I wanted to avoid using infinite loops because of the possibility of memory leaks (I believe allowing the process to exit totally frees up memory associated with that process and prevents leaks). I have not tested any of this, it's just a concept at the moment.

Something I would prefer to do would be to have one script be the controller of the rest of the scripts and have the supervisor only look after the controller. This saves me having to update the supervisors config file every time I add to the Python system.

The basic idea for the controller would be to have a loop fork new threads for each separate script/process and then have the controller wait for all the other threads to complete before allowing the loop to do another iteration (starting everything again).

I'm guessing what I need to do for this would be to wrap all the code in each script in a function, then import them as modules into the controller and call each function on a new thread inside the 'main' loop of the controller. Then the loop just runs, calls each function on a new thread and waits for each thread to finish before starting again.

How does this sound as a concept?

The only direct coding question I have is about turning my scripts into modules, how do I go from totally procedural code (no functions, no classes) into an importable module? can I wrap the procedures in just a function, or do I need to have them in a class also?

Answer 1

I have a similar system but have a different concept:

• I use cron to auto-start processes
• Check if related process is running
• If it is not running, then start the process.

The module uses psutil package to get a list of running processes, search for the related process and returns whether is it in the list or not. That may sound practical or not according to your use case though:

import psutil

class ProcessControl(object):
    """
    This program checks whether the given python file is running or not. If check_params flag is set as True, then
    it will be checked if an instance of the python file is running with all of the given paramters. If check_params
    flag sets as False, then the check will be made with only using the python file name and instances of the file
    running with different parameters will also be count.
    """
    def __init__(self, filename, *args):
        self.filename = filename
        self.args = args
        self.arg_num = len(args)

    def process_count(self, check_params):
        process_count = 0
        # Examine the process list to check if given process is running or not...
        for _prs in psutil.process_iter():
            try:
                _cmdline = _prs.cmdline()
            except TypeError:
                _cmdline = _prs.cmdline
            if len(_cmdline) >= 2 and "python" in _cmdline[0] and self.filename in _cmdline[1] and _prs.is_running():
                # We found an instance of the process that is running. Since This control function is triggered when
                # we run the python code, There would be at least one process (this one) which is running. Counting 2
                # or more processes means there is another instance which is still running when we trigger the python
                # code file
                if check_params:
                    # We also will check the parameters for the complete similarity
                    if len(_cmdline) == 2 + self.arg_num and all(str(_arg) in _cmdline[2:] for _arg in self.args):
                        process_count += 1
                    else:
                        # This is no match...
                        pass
                else:
                    process_count += 1
            else:
                # This is no match....
                pass
        return process_count

    def is_running(self, check_params=True):
        return self.process_count(check_params) > 1

I have little python files which have similar code as below:

Ex:

myFile.py

class MyCodeClass:
    def run_code(self):
         ...


if __name__ == "__main__":
    my_code = MyCodeClass()
    try:
        if ProcessControl(__file__).is_running():
            print "Already Running"
        else:
            my_code.run_code()
    except Exception as e:
        print e

And finally I have lines that trigger this file in my cron:

* * * * * python myFile.py

Logic:

cron triggers this file every x minutes. ProcessControl checks whether file is running or not. When calling is_running method, you can pass a bool value so the controller will ignore the parameters passed while running the python file. Like if check_params is True following two commands will be accepted as different and both will be triggered:

python myOtherFile.py 127.0.0.1 220
python myOtherFile.py 127.0.0.1 250

and disabling check_params will evaluate the second call as the same program and will not trigger it

Answer 2

I think you are reinventing the shell. Just make up a bash script that would control your modules

I would start up the scripts and allow them to run for a certain time with a finite loop (in each script) and then they would exit, then the supervisor would restart them every time they exit and keep them alive

This is better done with shell script rather than threading/forking manually

UPD: take a look at this Q/A at stackoverflow:

until myserver; do echo "Server 'myserver' crashed with exit code $?. Respawning.." >&2 sleep 1 done