I have a trained (e.g. fitted) machine learning model from sklearn in python. The model was fit using a fairly small dataset (~200,000 points) so I could easily fit the training data on a single machine. However, the model needs to be applied to a many other files, which in total comprise about 1.5 billion rows. I have access to a Spark cluster, but I do not know how to apply an sklearn model, since Spark cannot parallelize dataframes other than Spark dataframes.

Does anyone know how I could go about applying this sklearn model to this large amount of data in parallel?

More Details and Similar OTHER Questions
(to head off responses that do not apply to my particular problem)

If you have access to a Spark cluster, why not just use Sparks ML library?

Sparks ML library does not support the type of regression I am using, specifically Nearest Neighbor Regression using ball_tree. There are other regression/classification methods as supported by sklearn and not Spark that I want to use.

Similar Questions

https://stackoverflow.com/questions/47980936/train-multiple-different-sklearn-models-in-parallel (This question is about how to generate a model in parallel, not how to apply one;this also did not get into the guts how how to do it)

(Olivier Grisel's answer might be the beginings of the correct path for me, but I was unsure how to apply it to an existing Spark cluster. Furthermore, his answer is still about specific sklearn functions. I want to more generally apply a specific sklearn method across multiple dataframes/files. That is not the same thing)

  • 1
    Why the downvote? This is an intelligent, well thought-out question. It's just plain rude to simply downvote what is clearly a not-spammy question without more feedback. – Josh Dec 27 '18 at 15:36
  • People have voted to close this question, citing that assistance with writing or debugging code as being off topic. It's hard to tease out a conceptual software engineering question when it centers around a specific implementation of machine learning (sklearn, Spark). I think that may have been the reason for the down-vote (but I'm not the down-voter). – Greg Burghardt Dec 27 '18 at 17:45
  • There is no code; so that objection is incorrect. More generally, where should this question go? When I tried to put it on stackoverflow, I was directed to put it here by the new system that helps users put there questions where they are most appropriate. – Josh Dec 27 '18 at 20:46

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.