Machine Learning in Real Time

Question

I've been going over some tutorials on machine learning using Python and libraries including SciKit Learn and Tensor Flow. (Basic tutorials like creating an algorithm to predict a price given input values).

I've found that these tools are extremely powerful, but they also appear to be quite slow, and require a lot of tweaking due to overfitting or underfitting the data.

This wouldn't be a problem if you were dealing with a batch of data (such as you receive a new set of data once a day that you want to calculate predictions on), but I see this being very difficult to run something in real time. I know that there are many sites and applications that are executing machine learning algorithms in real time, so I'm trying to understand how they realistically accomplish it. A few possibilities I can think of are:

1. They use these ML tools (such as Python, SciKit Learn, Tensor Flow) in real time to predict real time values and continually learn, but require a huge amount of computing power and optimization to keep performance at acceptable levels.

2. They use these ML tools for real time predictions, but only update the algorithm periodically in some form of offline or batch process. This still requires a lot of computing power and optimization for the real time predictions, but not nearly as much as option 1.

3. They use these ML tools offline to figure out the algorithm, and then translate that algorithm into some other language better suited for their need (i.e. Java, C++, JavaScript, etc.). This would seem very tedious to do.

Or is there a 4th option that I'm missing?

Answer 1

I don't know how"they" do it, but I have applied ML to a real-time problem (spam mail detection based on data available in the SMTP dialog, i.e. before the actual message is transmitted.)

One thing is that training and applying a neural network are fundamentally different operations.

• Training is iterative and requires careful selection of network architecture, training algorithms and parameters, so it does indeed take a lot of time, and with more training samples this time increases a lot. This makes real time training unfeasible in my case and probably in most others.
• Applying a trained network to a set of input data can be done in essentially constant time, as it requires only a fixed number of multiplication, additions, activation function applications and comparisons to get the result. Some data preparation that's not constant is needed, too, but that doesn't make a significant difference.

So for my special case, your second option applies, and boldly extrapolating from a single sample I'd guess many real-time applications have similar characteristics.

However, it is entirely possible that organizations with much more experience and resources choose different options to achieve better results (especially to better react to changing situations.)