How would you implement a execution graph like dependency for python scrips?

Question

I am often confronted to this problem in my scripts and I feel I am lacking some software design culture, I hope this is the right place to ask and sorry if this sounds too simple.

I write many scripts in Python to do various ML related tasks. They start small (e.g. I just want to compute some results) and often end up doing many optional things (open data from disk or from remote db, compute stats, save results in this or that format, interactively view results, maybe also save snapshots).

So although the script in itself does not do much, it can still have many execution paths because of the different choices than can be made. For example, a script could have this components:

   read_from_db ---┐                      ┌-- view_results --- write_movies
read_from_files ---┴-- compute-results ---┴-- write_results

Some components are alternatives (reading occurs either from disk or from db) while others are not (results could be viewed and saved at the same time).

I put the reading part into different generators and pick the correct one depending on the options, I think this is fine. However the rest of my code is usually a bunch of ifs depending on the presence of certain flags:

while sample in sample_generator:
  res = compute_results(sample)
  if view:
    view_handles = view(sample)
    if make_movie:
      make_movie(view_handles)
  if save_results:
    save_results(res)

It is manageable for a small number of options but it tends rather quickly to be a mess when options are added up.

Because generators are a good answer to pack the loading part, somehow it seemed logical to me at some point to use coroutines to pack the other parts and link them together depending on the options, but the process of gluing the coroutines together is not very pretty and more importantly, it is not as easy to implement dynamic linking that could be useful in some cases (e.g. switch off visualization manually after reviewing a few cases).

I was thinking maybe using a signal/slot pattern would work, but isn't that overkill for this kind of script? How should I do to articulate dependent modules in that kind of situation? And how data should be shared between those components? (e.g. make_movie will take graphical components from view but in practice may also need information from say sample to build a name for the different movies).

Answer 1

Sounds to me like you're describing an ETL. You load some data, do some intermediate computations and go about storing it. Indeed this gets out of hand when the workflow gets complicated or the data too big.

It really depends on your systems needs - how critical it is to your organization, how stable do you want it to be and how important is visibility into that workflow is (its almost always a big plus).

Since you're familiar with python, you could look up Apache Airflow which is an orchastration platform in which you define the steps of your workflow (called operators by Airflow) and the relationships between them. It's quite verbose and might be an overkill but yet it's open source.

If this is something that's expected to stay small, you could wrap those main functionalities into methods - then you could push those conditions further down the stack, decouple your code and have it more readable:

def extract(from='db'):
    if from == 'db':
        yield ...
    else:
        yield ...

def transform():
    compute compute compute ...

def load(view=True, make=True):
    store()
    if view:
        view()
        if make():
            make()


while batch in extract():
    result = transform(batch)
    load(result)