3

I have an asm/C code which implements some image filters. The objective of this project is experimentation with different implementations, benchmarking, plotting and reporting data to write a paper with relevant insights.

So for example one work-flow I need would be:

  1. Start at a base code implementation (Step 0)
  2. Compile, benchmark, plot
  3. Change parameter X from 110 to 100 in main.c (Step 1)
  4. Compile, benchmark, plot
  5. Change parameter Y from 10 to 15 in main.c (Step 2)
  6. Compile, benchmark, plot

But I need to be able to access every step of the way with a git tag/branch/commit/something. So that I might re-run the test on a different PC or make changes to each step. For example I might decide that changing X from 110 to 100 was not enough, so I change step 1 to be change X from 110 to 80 for step 1.

I thought about using 1 branch for each experiment with a tag for each step, for example with the tags being step0-2 and the branch being experiment1:

git checkout experiment1 #step0
compile, benchmark, plot
git checkout -b test step1
compile, benchmark, plot
git checkout -b test step2 #overwriting test?
compile, benchmark, plot
git branch -d test

But with this solution I can't easily change step 1, I'd need to start from the base and do it all over again, creating a new experiment branch (because I shouldn't commit between commits right?).

I also heard git branches should not be used for things not meant to be merged back. Am I over-complicating things? Is there a more obvious/simple way of managing this? Is the answer not git and I should use some other system?

  • If a change in step 1 turns out to be too little/too much, do you really have to go back to step 1 and change that, or could you just as easily create a new step to adjust parameter X from 100 (end-state of step 1) to 80? – Bart van Ingen Schenau Oct 2 '16 at 6:55
  • It's the same, since I still have to put the change between step 1 and step 2, if I put it before or after or step 1 I think it's the same. – Damian Pereira Oct 2 '16 at 6:57
  • Let me rephrase my question: does the change have to be between step 1 and step 2, or could you also apply the change after step 2? Especially if you find that step 1 wasn't good enough after you already applied step 2. – Bart van Ingen Schenau Oct 2 '16 at 7:43
  • But if I apply it after step 2, then I can't test step 1 without step 2. Because I can choose between step1, step1+step2 and step1+step2+step1change. Can't choose step1changed easily. Anyway I'll take the advice from the answer and make some changes, if it works I'll answer my own question. – Damian Pereira Oct 2 '16 at 19:01
4

You shouldn't use git to handle this complexity. It would be a nightmare, the branches would diverge to far apart with time. You would be left with snippets of code that would all need to be separately managed.

A much more flexible solution would be to make the varying parts of your code base flexible. For instance, if you need the ability to change input parameters, make the parameters configurable on the command-line. This requires one binary, but you can run multiple tests.

If you need to implement separate algorithms/functions, you can make them configurable as well. Function pointers in C are a great way to do this, and this can be configurable from the command line also.

| improve this answer | |
  • I used the parameter option for simplicity, but the changes range from "changing 1 value in one line" to "Completely change the whole code of this filter". The biggest problem is I can't compromise past test results by making changes to the base code with each new experiment, so I'd still need to keep separate revisions somehow, I need to be able to go back to the pristine exact configuration in all variables, for consistency. – Damian Pereira Oct 1 '16 at 22:53
  • 4
    @DamianPereira: "Completely change the whole code of this filter" can be handled by "add an additional filter (lets call it variant 2 of filter foo1), and add a parameter to switch between foo1 & foo2 at run time". Also, make sure your programs makes a detailed log containing the exact parameters it was called with in each run. – Doc Brown Oct 2 '16 at 7:16
  • The problem is suppose I have data for experiment 1, then for experiment 2 I add another filter and switch with function pointers. Then data for experiment 1 is sorta compromised. I mean yes it should not change at all, but all variables should be controlled. Ideally I'd have the same code, and be able to add more experiment without changing 1 line of the original experiment environments at all. – Damian Pereira Oct 2 '16 at 19:05
  • 3
    @DamianPereira I'm not sure I follow your idea. The data for experiment 1, would not be "compromised" by running your binary a second time with another data set? If you control the function pointer with a command-line switch, you should easily be able to reproduce both sets of data output, the only difference would be the command line syntax. – nullverb Oct 2 '16 at 20:29
  • But I'd have to change lines of code in files common to both experiment, I know it should not affect anything, but it could get messy. For example I add a switch in the main file for experiment 2, and gcc decides to optimize the file differently and it doesn't fit whole in the cache somehow or something unexpected happens. That would change the results of experiment 1 if I wanted to re-run it and get the same numbers. It's a very slim chance and mostly theoretical problem, but I'd rather be sure, scientific method and all that, – Damian Pereira Oct 3 '16 at 21:07
0

Old question, but an addendum answer.
There is an approach used for software "experiments" which I have found very useful. The article is "A Git workflow for code experiments", by Steve Brudz: https://medium.com/defmethod-works/quick-tip-git-workflow-for-code-experiments-82af10b1c5c4

Perhaps it could serve as a base for a more relevant answer to the question?

| improve this answer | |
0

Separate code, configuration, and executions.

For larger experiments (weeks of experimenting vs. hours or days of experimenting). And running and develloping the experiments in batch (vs. a more interactive approach). I would suggest several separate repositories:

  1. The codebase of highly configurable code.
    Using your normal git branching practice.
  2. The parameters of the experiments

    • a config file(s) that the codebase understands
    • minimal version of the codebase that can run the config file; for example via a git submodule.

    Git history in this repo follows your lines of reasoning. It may contain many branches, that will not be merged. The history is less about reusing config files, the focus is on your line of thought.

    The ability to git diff between versions and branch can come in handy when diagnosing the cause of differences in behaviour between experiments.

  3. The results of the experiments.
    Including:

    • The configuration parameters of the experiment (again via a submodule).
    • The actual version of the codebase that ran the experiment (submodule).

    Git is not the ideal tool for this kind of data, but it works and has the advantages that software developers (including OP) are familiar with it.
    Git's history is not that relevant. Diffing can help, if the nature of the experiments and their results allows it. Branching is not that relevant, perhaps use just a linear single master branch history. Use the commit message to identify and find the right experimental results (see git log --grep=... https://stackoverflow.com/a/7124949/814206)

Optionally you may have:

  1. Report
  2. Source data the experiment should process.
| improve this answer | |
0

For small experiments (hours to days of trying things) in an interactive setting (a REPL or quick edit, compile, run loop) you should strive for two things:

  1. Not disturb your flow of experimenting, making changes, and trying again; and,
  2. keep track of what you did.

For "edit, compile, run"-loops (ab-)using unit tests works well:

  1. Create a new test method with a name containing a short description of your experiment, a sequence number, and your initials.
  2. Write the experiment as a unit test method, for now, possibly without assertions or with wrong assertions.
  3. Compile & run, many IDEs allow quickly running a single test method in their unit test framework.
  4. Document the outcome of the experiment via assertions (adding or changing those from step 2). Resist the temptation to alter the experiment.
  5. Commit this experiment to git (single branch).
    (Pull the experiments from your team mate if needed. Hence the initials in the method name.)
  6. Rince and repeat (copying code from previous experiment test methods as needed)
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.