Let's say that I have a service like that takes in some text, classifies it, and then outputs the classification. The critical path between entry and classifier is as follows:

Query -> QueryPreprocessor -> ML Model

Now, I've obtained some ground truth on a random set of queries, and I want to retune the classifier based on the ground truth. What's the best way to do this retuning given we have a QueryPreprocessor module in between the Query and the ML Model?

I've thought of a few options, but not sure which is the best approach to take -

  1. Write an offline program that imports the QueryPreprocessor, runs it on the Queries, Passes it to the ML Model with different params, and determine the best performing config.
  2. Have the ML Model log the incoming requests, and outgoing responses, and then just tune on those inputs / outputs.
  3. Bring up multiple dev instances of the service at different fixed tuning points, and run it on the full data set.

All of these seem to have pros and cons. A few that I thought of as follows:

  1. This has the pro that it's relatively fast in terms of evaling, however it feels like we'll shoot ourselves in the foot down the line. We're putting a direct dependency on a serving-time module which means that serving-time params aren't going to be necessarily captured (this can be addressed if we fully capture all such serving-time params). In addition, for every new module that is added upstream of the ML Model, this eval code will ned to be updated.
  2. This seems like the best approach to me, however if the captured queries are "stale" (i.e. the upstream modules changed), then even after tuning correctly, it's possible that the results won't translate to production.
  3. This has the pros that it runs the full e2e stack as it is today, however it feels super heavy weight and resource intensive for ML developers to do. In addition, doing fine-tuned tunings isn't possible without potentially spawning a lot of instances.
  • I first read the title to be "best way of tuning an evil ML model" .. lol
    – Peter M
    Commented Sep 27, 2021 at 22:25

1 Answer 1


It's one thing to train a model, and it's another to offer it as a predictive service. Training code looks significantly different to prediction code. In my opinion, you should appropriately split these tasks up.

The model currently in the service (production) should be used for prediction only, and never directly trained. Instead, you train another model in a separate environment (development), which replaces the one in the service once it reaches a better performance.

These are the steps I would take:

  1. Figure out the performance of the production model.
  2. Copy the model to development, along with its parameters and weights.
  3. Test your development model with all inputs and parameters that you want, log the results.
  4. Periodically (hourly? daily? weekly?), check if one of the tested models in development outperforms the one in production. If it does, replace it.

Depending on the task, your prepreprocessing steps may also differ between model training and prediction. If that is the case, you may want to create dedicated methods in QueryPreprocessor for training/predicting, or split it into distinct train/predict classes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.