I have a trained (e.g. fitted) machine learning model from sklearn in python. The model was fit using a fairly small dataset (~200,000 points) so I could easily fit the training data on a single machine. However, the model needs to be applied to a many other files, which in total comprise about 1.5 billion rows. I have access to a Spark cluster, but I do not know how to apply an sklearn model, since Spark cannot parallelize dataframes other than Spark dataframes.
Does anyone know how I could go about applying this sklearn model to this large amount of data in parallel?
More Details and Similar OTHER Questions
(to head off responses that do not apply to my particular problem)
If you have access to a Spark cluster, why not just use Sparks ML library?
Sparks ML library does not support the type of regression I am using, specifically Nearest Neighbor Regression using ball_tree. There are other regression/classification methods as supported by sklearn and not Spark that I want to use.
https://stackoverflow.com/questions/47980936/train-multiple-different-sklearn-models-in-parallel (This question is about how to generate a model in parallel, not how to apply one;this also did not get into the guts how how to do it)
(Olivier Grisel's answer might be the beginings of the correct path for me, but I was unsure how to apply it to an existing Spark cluster. Furthermore, his answer is still about specific sklearn functions. I want to more generally apply a specific sklearn method across multiple dataframes/files. That is not the same thing)