I am often confronted to this problem in my scripts and I feel I am lacking some software design culture, I hope this is the right place to ask and sorry if this sounds too simple.
I write many scripts in Python to do various ML related tasks. They start small (e.g. I just want to compute some results) and often end up doing many optional things (open data from disk or from remote db, compute stats, save results in this or that format, interactively view results, maybe also save snapshots).
So although the script in itself does not do much, it can still have many execution paths because of the different choices than can be made. For example, a script could have this components:
read_from_db ---┐ ┌-- view_results --- write_movies read_from_files ---┴-- compute-results ---┴-- write_results
Some components are alternatives (reading occurs either from disk or from db) while others are not (results could be viewed and saved at the same time).
I put the reading part into different generators and pick the correct one depending on the options, I think this is fine. However the rest of my code is usually a bunch of
ifs depending on the presence of certain flags:
while sample in sample_generator: res = compute_results(sample) if view: view_handles = view(sample) if make_movie: make_movie(view_handles) if save_results: save_results(res)
It is manageable for a small number of options but it tends rather quickly to be a mess when options are added up.
Because generators are a good answer to pack the loading part, somehow it seemed logical to me at some point to use coroutines to pack the other parts and link them together depending on the options, but the process of gluing the coroutines together is not very pretty and more importantly, it is not as easy to implement dynamic linking that could be useful in some cases (e.g. switch off visualization manually after reviewing a few cases).
I was thinking maybe using a signal/slot pattern would work, but isn't that overkill for this kind of script? How should I do to articulate dependent modules in that kind of situation? And how data should be shared between those components? (e.g.
make_movie will take graphical components from
view but in practice may also need information from say
sample to build a name for the different movies).