What is the best way to version control ML model development experiments and the production microservice code that serves the best model?
Currently, I have one git repository for ML model experiments, with branches for each experiment -- the 'research' repo and another git repo that stores the microservice code that serves the best model's predictions -- the 'inference' repo.
When a better ML model is discovered, this involves copying some of the ML code + weights from the research repo into the inference repo (earlier model code is overwritten). There might be some additional changes to the model to optimize for production use (e.g. quantizing, pruning, converting to ONNX format, etc...). Code that is strictly used during training/development is left behind and not copied over to the inference repo.
Concerns with the current setup:
- The copying ML model code step smells a bit like repeating myself (e.g. not DRY).
- One repo feels like a simpler way. However, if one repo was used for research and inference, then git clones would include all the research and experiment branches (which seems like a lot of unnecessary code for the inference microservice).
- Research and production seem like different concerns (thus the two repos) but the best model is a source of overlap between the concerns.
Looking for ideas for how to organize different model research experiments and production microservice code in git.