2

Background:

  • So there is a (Ruby) web app with a production Postgres DB (hosted in the cloud)
  • I would like to run some machine learning algorithms in a Python setting on the production data and (ultimately) deploy the model in a production setting (in the cloud)
  • I only know how to run these algorithms locally on say a Numpy array that fits in memory and assuming the training data is fixed
  • Let us say the dataset of interest would ultimately be too large to fit in memory, so the data would need to be accessed in batches.

My general question is:

What is a good way to go about setting up the pipeline to run the algorithms on the production data?

To be more specific, here is my current reasoning, that may or may not make sense, with more specific questions:

  • Considering the algorithms will need to access the data over and over, read speed will be pretty important. We cannot afford to access it over the network and cannot keep querying the web app production db anyway. What is the best way to store the data and make it available to the machine learning algorithms to process? Copy everything to another relational DB that the Python code can access locally?

  • Finding the right model is probably easiest if done locally on a sample of the data that fits in memory. Once a good candidate is found, we can retrain it, with all the data we have. Should we do the second step locally as well? Or you should generally try to setup a complete production pipeline that allows you to work with a larger amount of data at this stage already?

  • Let us say you have new data being written regularly. If you do the initial training by visiting batches of the data you have at time 0, and then stop training, you probably have to retrain it from scratch using all of the data you have at some later time t? Is the re-training something that is reasonable to automatize in production?

General hints and sources that help with these kind of questions are appreciated.

0

(Important Preliminary Consideration) Before attempting to answer the questions you asked, I would highly encourage you to consider renting a high-memory machine for the ML modeling tasks if you're already in the cloud and talking about batch jobs. A 500GB RAM box can be had for $5/hr these days, and it'll be worth every penny (if you have little enough data that such a machine suffices -- anything after 1-6TB RAM is probably pushing the limits of cost-effectiveness, and you'd probably be better off with a distributed system).

(The questions you asked) I'll focus on your last three bullet points here:

  1. It depends. If querying the production DB direction makes sense in your business and if the ML pipeline is closer to a streaming model so that each piece of data is touched few times then you might not need/want any layers between the DB and the ML batch processors. That said, lakes/warehouses/replicas and whatnot are common to isolate customers from the potential performance degradation from analytics. As to the performance question though, to a reasonable order of approximation the difference between "close postgres" and "far postgres" doesn't really matter unless you have synchronous round trips (and if you do...don't do that). There's a much bigger jump from "close postgres" to "local memory", and trying to frame the problem in a way so that you can load a blob of data into each ML machine, operate on the blob, and then aggregate the results with little to no back-and-forth will probably perform better than a solution involving a lot of shared state in a nearby database. I wouldn't be inclined to spin such a thing up for performance reasons (though exceptions obviously exist).
  2. It depends. The inferential pipeline absolutely needs to live in production. The model building pipeline can live anywhere. If your model auto-updates in response to new data I would propose it's easiest for the model building pipeline to also live in production. Otherwise it depends on other factors like how often you change models, how good your tests and metrics are, how critical time-sensitivity is, how static the data is, how critical accuracy is, the compute resources needed to train a model, your model version control process, and so on. If your model doesn't auto-update then where you train it doesn't matter all that much in most scenarios.
  3. Yes. People automate re-training all the time. There are even algorithms designed to auto-update in response to new data. Good performance metrics and monitoring are mostly non-negotiable in an auto-updating world.

(Desired Architectural Qualities) Unfortunately there isn't one correct answer to your question. The unique characteristics of the production system, the data you're working on, the ML models you're considering, and the dynamics of the underlying business will push you toward different solutions. Here are a few questions that might help point you to a good solution for your use case. Getting a good handle on requirements will allow you to take an informed and principled approach to choosing between different solutions:

  1. Will the production database suffice for your ML pipeline?
    1. Is there a high availability assumption for the DB's other clients?
    2. Can the machine handle the additional analytics load?
    3. Does the data already easily map via SQL to a form the ML pipeline can handle?
    4. Is the data especially time-sensitive?
  2. Do the ML pipelines need to be versioned and stored?
    1. Do you need to be able to roll back pipeline deployments?
    2. Do you want to be able to analyze the pipeline's progress over time?
  3. Should the ML pipeline be pre-trained or automatically trained?
    1. Is the data naturally incremental and read-only (e.g. most time-series data), or does the underlying model have a lot of updates and deletes?
    2. How problematic is it if the ML pipeline performs poorly?
      1. Consider various facets question of this including false positives, false negatives, implicit bias, worst-case scenarios, median-case scenarios, and so on.
    3. Can you detect and respond to a poorly performing model?
    4. Is the data sufficiently static that the model can change slowly or never?
    5. Do the questions being asked naturally map to a model designed to adapt to new data?
    6. Is deployment easy enough in your shop that you can easily switch to/from automatic/manual training if one approach doesn't work well?
  4. If an additional data store is needed, where should it be located?
    1. What are the data ingress/egress rates with your cloud provider?
    2. Are there ingress/egress exclusions when you stay in the provider's network?
    3. Does the provider offer everything you need to build the rest of the ML pipeline?
    4. Do you have the means to locally (with a broad definition of "local") host any portion of the system?
  5. Should the ML pipeline even be hand-built in the first place?
    1. Can the organization afford the ongoing hardware/software/data cost?
    2. Could the organization afford options like Google AutoML?
    3. Is the data especially secure or sensitive?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.