Perhaps I should set the correct expectations at the beginning: the best solution might not be object-oriented (OO). You should use what is the most appropriate, not what is the purest-OO idea.
Are you working mainly in Python, or one of the domain-specific languages (DSLs) used for neural network training?
You can try refactor the code from e.g. total 150 lines, breaking down into several functions. These functions are reusable across your variants of programs. Once you have refactored (extracted reusable functions), discuss with your team members so that they all know that you have provided these functions to them, so that they will use it. Reusable functions shared within a team requires a moderate effort in code maintenance.
Some techniques that are needed to solve the problem (such as "polymorphism", "interface/implementation", "higher-order functions" or simply
eval()), depend on whether they are allowed in that language. Most languages will try to provide at least one way to solve this particular problem, however DSLs might not provide any. (For example, GPU programming languages don't have any.) In that case, you may have to do part of the programming (the outer-loop) in Python, calling into the DSL for smaller pieces of tasks.
In other cases, you may have to use Python to generate source code for the DSL, by formatting and concatenating strings and writing them into a text file which can be used as a source file for the DSL.
To give more useful and specific answers, please try to explain your situation better. Try to explain with pseudo-code. It will also be language-specific. I would say that Python is designed to solve this particular problem for scientists better than other languages.
An example of loading and saving at intermediate checkpoints look like this:
// code for step 7
if (min_step <= 7) and (max_step >= 7)
if (input_for_step_7 == null)
if (step_7_input_filename == null)
throw error ("either need execute step 6 or need input file for step 7")
input_for_step_7 = load_input_for_step_7(step_7_input_filename)
assert(input_for_step_7 != null)
output_for_step_7 = execute_step_7(input_for_step_7)
end // min_step, max_step
In neural network training, "record and play back" are still useful, but have their limitations.
This is because modifications of one step of the algorithm will fundamentally change the characteristics of its output. Therefore, to test the effects of modifying one step of the algorithm, the entire chain of algorithm steps have to be re-executed. Thus, the idea of "testing in isolation" is only useful in a limited sense.