The Answer to Your Stated Question
Kubernetes does not inherently support parallelization of code executed within a container runtime.
Kubernetes core abstraction - the pod - enables you to spin up multiple copies of the same application on different nodes, but does not inherently coordinate across these applications for a single request. Kubernetes lets you spin up a
Service, which load balances requests across each pod - however, no abstraction in Kubernetes lets you send requests to more than one pod at a time or "partition" requests so that they are sent to all pods at once.
That being said, Kubernetes does support coordinating across pods through means of stateful sets and operators. Neither are particularly easy to work with and are designed with the use case of managing distributed databases in mind, which require long-lived persistence and a notion of leader election. Neither feature solves what you're trying to do.
The Answer to Your Actual Problem
In my opinion, you are trying to solve your issue at the wrong layer of abstraction. Kubernetes and Docker are designed to simplify deploying and managing reproducibility across environments for your application. They don't solve your fundamental problem, which is pure performance .
Here are some obvious points I'm seeing:
You're doing a brute-force search across all of Y for every term found in X, but what you're trying to do is essentially what's known as full-text search. Why not take a leaf from the people who solved full-text searching and generate an inverted index?
In other words, you would preprocess dataframe
Y and generate a map
Z from words in
Y to location in
Y. Then you just check if your search term in X exists in
Z - if it does, great, you've found your location in
O(1) time (assuming preprocessing
Y to form
Z is cheaper than searching over all of
Y for every term in
X, which it should be since you're only doing it once).
If that approach doesn't yield success, you can still "buy" your way out of the problem by using a
MapReduce implementation such as Hadoop or Spark (the latter of which I strongly recommend on account of native Python bindings) and putting your 2-3 nodes to good use.
Unlike Kubernetes, this actually does solve the problem of partitioning
Y across multiple nodes, having each of them check if a term from
X lives in their partition, and then combining the results of each node's search.
- If that still doesn't yield success, it may be time to ditch using
pandas altogether and use a database with fast full-text search enabled. The logic here is that you join all your multiple sources together from what used to be
Y, store it in a production-ready database, and then query the database for all instances of your search term.
These are all suggestions and your mileage may vary, but based on your description, you appear to be asking a classic XY problem and I hope this gives you some context of why Kubernetes is the wrong tool for what you're trying to solve here.