Let's say that I have a service like that takes in some text, classifies it, and then outputs the classification. The critical path between entry and classifier is as follows:
Query -> QueryPreprocessor -> ML Model
Now, I've obtained some ground truth on a random set of queries, and I want to retune the classifier based on the ground truth. What's the best way to do this retuning given we have a QueryPreprocessor module in between the Query and the ML Model?
I've thought of a few options, but not sure which is the best approach to take -
- Write an offline program that imports the QueryPreprocessor, runs it on the Queries, Passes it to the ML Model with different params, and determine the best performing config.
- Have the ML Model log the incoming requests, and outgoing responses, and then just tune on those inputs / outputs.
- Bring up multiple dev instances of the service at different fixed tuning points, and run it on the full data set.
All of these seem to have pros and cons. A few that I thought of as follows:
- This has the pro that it's relatively fast in terms of evaling, however it feels like we'll shoot ourselves in the foot down the line. We're putting a direct dependency on a serving-time module which means that serving-time params aren't going to be necessarily captured (this can be addressed if we fully capture all such serving-time params). In addition, for every new module that is added upstream of the ML Model, this eval code will ned to be updated.
- This seems like the best approach to me, however if the captured queries are "stale" (i.e. the upstream modules changed), then even after tuning correctly, it's possible that the results won't translate to production.
- This has the pros that it runs the full e2e stack as it is today, however it feels super heavy weight and resource intensive for ML developers to do. In addition, doing fine-tuned tunings isn't possible without potentially spawning a lot of instances.