How to maintain well structured code and latency optimized code together?

Question

I'm working on a c++ data analysis project. My workflow goes like this

1. Analyze the data and build models
2. Optimize the code for latency, to deploy for production
3. goto 1

Step 1 has lots of machine learning parameters, using which I test very minor variations of algorithms. In step 2, I clean up the unused parts of the code (non optimal parameters), optimize code for latency (changing maps to arrays for example), and deploy the code. These modifications are done directly on the step 1's code. No separate branch in maintained.

When new data is obtained and step 1 is required to be repeated, I would have lost the ability to test minor variations of an algorithm. One way to solve this is to maintain two branches. One will be an experimental branch, which has all the parameters for the minor variations of the algorithm. Another branch will be latency optimized code. But, the problem here is any small change in experimental branch will need to be repeated in the latency optimized branch, because there two branches cannot be merged. There are huge differences (even new files appearing) between experimental branch and latency optimized branch, which hinder direct merging.

Is there any other way to solve this?

EDIT1: Another example of step 2

For the sake of illustration, lets say step 1 leads me to a predictor y = f(x) = floor(x^3 + 3x^2 + 5x), where floor(z) = (value of integer closest but <= z). x in [0, 100]. The basic way to make prediction (in deployment) is to evaluate f(x). But, observe that the whole function is discrete in output and increasing. So, another way to predict is to store ranges of x, which map to y = 1, 2, ..., and do binary search on it. This will lead to maintaining a vector with entries, {(lbx_1, y_1), (lbx_2, y_2), ...}, where y_i = i, lbx_i gives the least value of x such that f(x) >= y_i. In this case, for any input, a simple binary search will fetch the prediction much faster. (In practice, f(x) is much more complicated than the above mentioned function).

The latency optimized branch will have this map for prediction. The experimental branch will evaluate the function. But I also need the general evaluator for experimentation, in case my predictor turns out to say y = f(x) = (x - 1)^2.

Answer 1

I would not try to abuse a version control system like git for this.

For complex optimization as described here, which can only be done manually, it is probably best to keep both versions of a function in the same code base in parallel. Both, the non-optimized version and the optimized one should be placed somewhere near to each other. In C++, switching between both versions could be done, for example, by using the preprocessor, or by a run time mechanics. (Template metaprogramming zealots will surely prefer to use partial template specialization.)

For example

 int f_slow(int x) {return floor(x^3 + 3x^2 + 5x);}

 int f_fast(int x) {/*  50 lines of optimized code*/}

 #ifndef OPTIMIZED
 #  define f f_slow
 #else
 #  define f f_fast
 #endif

This way, f_slow is kept as a way more readable version of f_fast which serves as documentation. You should also use this for automated tests, asserting f_slow(x) == f_fast(x) for a range of test values, so making sure your optimization does not introduce some unexpected bugs.

Of course, this does not take the burden from you of maintaining two versions of the same function. But I think this is more or less unavoidable, since writing an optimizer seems to be pretty infeasible for this task.

The most effective course of action in this situation is probably to apply some self-discpline and work within clearly defined phases:

• an experimental phase, where only f_slow exist, and where you are modiyfing it quickly until it suits your needs

• a phase where f_slow is kept unchanged, and f_fastwill be created, including the tests

When it comes to the point where you are going to do you experiments with a new version of f(x), give it another name (like f_slow_v2), keep f_slow as it is, so you can use f_fast as a blueprint for f_fast_v2. Later, when you are sure neither f_slow nor f_fast are needed any more, you can remove them both from the code simultanously.