0

I'm currently working in an environment where I have the following:

  1. Multiple Inputs (with multiple versions)
  2. Source code to generate output (multiple versions)
  3. Output generated by a combination of input and source code

The top level directory looks like this:

inputs/
outputs/
src/

I'd like to somewhat generalize this concept and create a script that helps me automatically keep track of the information needed to generate a given output.

Thus far, I have settled on the following hierarchical structure to help me do that:

inputs/

I will keep track of each discrete set of inputs in a folder. While using Git is potentially possible, tracking large, possibly binary files still doesn't seem feasible in Git. So, for example, I will set up this directory structure as such:

inputs/inputs-1-v1
inputs/inputs-1-v2
inputs/inputs-1-v3
inputs/inputs-2-v1

src/

Source code, version-controlled by Git. These are meant to represent distinct experiments or analyses, and can be in any arbitrary language. Example:

script-1
script-2

outputs/

Suppose inputs/inputs-1-v1 is given as input to script-2. Then, the resulting output folder would be generated:

outputs/script-2/git-hash/inputs-1-v1

This structure is flexible and I'm just thinking it through now, but not really why I am asking my question. I figured it was necessary to give some background.

Question(s)

Presently, I am planning on writing a "master" script that I can use for this generic project architecture, that I can run from the top level directory:

run -c "script parameter 1 parameter2" -i <inputs folder>

This would result in a simple command expansion to the following:

src/script parameter1 parameter2 -i ../inputs/input-folder -o ../outputs/script/git-hash/input-folder > ../outputs/script/git-hash/input-folder/stdout.txt 2> ../outputs/script/git-hash/input-folder/stderr.txt

However, this feels very clunky to me. It forces my scripts to expose a CLI that accepts the -i and -o arguments. It evokes the question why I would write such a master script in the first place, but I feel like abstracting out the idea of creating these output folders is a good plan, rather than repeat that logic in a number of separate scripts.

I think what bothers me most is the lack of any declaration of a formal interface. I am requiring the implementer to add these -i and -o options to their scripts. If this was a Java class for example, I might create an Experiment interface and and have script1 implement it.

What I intuitively feel should happen is something more Unixy, where I could pipe input from the input folder and simply redirect the output of the script without having to have the script write files explicitly. However, this is complicated by the fact that the script might write several files (image files, text files, etc.) as well as read from several files for a given input.

So, in summary, I am asking:

  • What are some other possible approaches here?
  • Is the arbitrary language constraint on the scripts preventing better design?
  • From a software engineering perspective, what constraints could/need I relax to make this design better?

I found the following related question, but that is more asking about structure.

https://academia.stackexchange.com/questions/36995/how-to-keep-code-and-output-organized

Edit: Perhaps the following question is clearer: what is the best way for me to communicate the fact that the subordinate script should write its output to a certain directory? Should the script even know it where it should write its output, or would it be better to try to write it to something like stdout? The approach I describe above requires the user to know that their script needs the explicit interface defined above, and I don't know if that is good design.

  • Consider using a good version control system like git which solves half of your problems. – Basile Starynkevitch Aug 23 '17 at 19:17
2

In my opinion, here are the answers:

  1. What are some other possible approaches here?

    • As long as inputs and outputs are text files, there are no difficulties in versioning them with the code, therefore I'd do this in order to remove files like "input1-v1" and "input1-v2";
    • Having that said, I would provide a makefile in the project's root, which will have all commands (the run of your master script would be a target in this makefile); the run target would receive as parameter which input you are processing, and then the makefile just passes this parameter to the main source-code (compiled through the makefile if necessary);
    • after the processing of all inputs, you version the outputs and commit everything to the server, creating a TAG in order to report all experiments to a specific version
  2. Is the arbitrary language constraint on the scripts preventing better design?

    • If you use the makefile or other similar approach, language or SDK should not be a problem, even OS should not be problem;
  3. From a software engineering perspective, what constraints could/need I relax to make this design better?

    • First, we need to understand your concerns, needs and constraints; from what I got, I suggested the above steps to fulfill the versioning requirements; I wouldn't change any constraint based on what you described in your question.
  • I suppose my primary concern is that I am requiring the implementer to add these -i and -o options to their scripts. With the make approach, I could replace placeholder text in the files with the filenames, as you suggest, and this does seem to be a cleaner solution, although it still places the burden on the programmer to have these placeholders explicitly coded in their script. I am starting to feel that the only way to get around this issue, though, is to make use of stdin and stdout and make my script much more Unix-like and modular. – faulty_ram_sticks Aug 23 '17 at 19:23
  • 1
    You could provide helpful information in your makefile (in a 'help' target eg.) explaining all targets available and that the user must provide the input file in a specific folder with a specific pattern (eg: input-<index>), and for each specific target, you only pass the <index>, and the make would concat it with 'input', and then redirect the stdin to point to the file, and stdout to point to a specific output file, and just call the script. Does this approach works for you? – Emerson Cardoso Aug 23 '17 at 19:38
  • 1
    By doing the above, the amount of changes in the makefile will be much smaller than in the source scripts. – Emerson Cardoso Aug 23 '17 at 19:40
  • Yes, I think that that approach would work. I think the crux of the issue is that I want to be able to have a script read a file without explicitly coding the name of the file into the script, and for that, I think that stdin really is the only answer. Here, make can serve to coordinate that activity. I think that such a setup exposes a flaw in my own scripts, which can have multiple (binary) outputs. In such a situation, stdout does not work well, and the issue of "do one thing, and do it well" comes into play, but this can be remedied by refactoring. – faulty_ram_sticks Aug 24 '17 at 4:03
1

Encode the input/output file naming scheme in the program

Consider just baking the input/output file naming scheme into the program. if you do that you, then your arguments will specify the input directory; the path to the output directory will be automatically derived from the base output directory and the input directory name. The base output directory could be hard-coded, or relative to the input directory, or relative to the executable. You might allow an argument to override the base output directory for testing.

Write a little script for each input directory

In this approach, you create a little script for each input directory. That script encodes the output directory. This replaces the logic you are envisioning with you just creating a script and typing in the right directory. This turns output directory names from "fully automatic" to "semi automatic": You manually configure the output directory just once, when you create the script.

Use make

In this approach, you use make to run the program (and possibly to build it as well). Make encapsulates the rule to derive the output directory from the input directory. If you like, you can configure make to rerun the program when the program changes, or when the input data changes. You can create a rule that runs the program against every input directory, if desired.

This is my favorite approach: It's sort of what make was designed for. Although its DSL can be a little cryptic, it works very well for things like this.

  • So, I believe the first option you provided is what I was presently planning on. Inside the "run" master command, I hardcode this logic, and output dir is derived from the input dir, unless you mean that each script should hardcode this logic. Would you mind updating your answer to provide examples of the workflow or commands that might have to be run for the second approach? I hadn't thought of using make, but it seems that addresses a somewhat different issue than what I was trying to solve. I'm mostly concerned about the best way to communicate the directory information to the scripts. – faulty_ram_sticks Aug 23 '17 at 16:48
  • Unless I've misunderstood you, this is exactly the sort of problem that make was designed for. However, I've failed to address the versioning part of your question; @emerson Cardoso'a answer does address that and is a better answer. – Wayne Conrad Aug 23 '17 at 19:37
  • Thanks for your help. @Emerson Cardoso's answer helped me understand more of what you meant here, and I agree, learning more about it, make does seem to be perfect for coordinating this kind of setup. – faulty_ram_sticks Aug 24 '17 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.