I'm currently working in an environment where I have the following:
- Multiple Inputs (with multiple versions)
- Source code to generate output (multiple versions)
- Output generated by a combination of input and source code
The top level directory looks like this:
inputs/ outputs/ src/
I'd like to somewhat generalize this concept and create a script that helps me automatically keep track of the information needed to generate a given output.
Thus far, I have settled on the following hierarchical structure to help me do that:
I will keep track of each discrete set of inputs in a folder. While using Git is potentially possible, tracking large, possibly binary files still doesn't seem feasible in Git. So, for example, I will set up this directory structure as such:
inputs/inputs-1-v1 inputs/inputs-1-v2 inputs/inputs-1-v3 inputs/inputs-2-v1
Source code, version-controlled by Git. These are meant to represent distinct experiments or analyses, and can be in any arbitrary language. Example:
Suppose inputs/inputs-1-v1 is given as input to script-2. Then, the resulting output folder would be generated:
This structure is flexible and I'm just thinking it through now, but not really why I am asking my question. I figured it was necessary to give some background.
Presently, I am planning on writing a "master" script that I can use for this generic project architecture, that I can run from the top level directory:
run -c "script parameter 1 parameter2" -i <inputs folder>
This would result in a simple command expansion to the following:
src/script parameter1 parameter2 -i ../inputs/input-folder -o ../outputs/script/git-hash/input-folder > ../outputs/script/git-hash/input-folder/stdout.txt 2> ../outputs/script/git-hash/input-folder/stderr.txt
However, this feels very clunky to me. It forces my scripts to expose a CLI that accepts the -i and -o arguments. It evokes the question why I would write such a master script in the first place, but I feel like abstracting out the idea of creating these output folders is a good plan, rather than repeat that logic in a number of separate scripts.
I think what bothers me most is the lack of any declaration of a formal interface. I am requiring the implementer to add these -i and -o options to their scripts. If this was a Java class for example, I might create an Experiment interface and and have script1 implement it.
What I intuitively feel should happen is something more Unixy, where I could pipe input from the input folder and simply redirect the output of the script without having to have the script write files explicitly. However, this is complicated by the fact that the script might write several files (image files, text files, etc.) as well as read from several files for a given input.
So, in summary, I am asking:
- What are some other possible approaches here?
- Is the arbitrary language constraint on the scripts preventing better design?
- From a software engineering perspective, what constraints could/need I relax to make this design better?
I found the following related question, but that is more asking about structure.
Edit: Perhaps the following question is clearer: what is the best way for me to communicate the fact that the subordinate script should write its output to a certain directory? Should the script even know it where it should write its output, or would it be better to try to write it to something like stdout? The approach I describe above requires the user to know that their script needs the explicit interface defined above, and I don't know if that is good design.