Skip to main content
Software Engineering

Return to Question

changed tags
Source Link
calin.bule
calin.bule
  • 3
  • 2

I am fairly new to Docker and Kubernetes and I have a question for which I could not figure out the answer myself. I am working on an application that does string matching on data extracted from multiple sources but the problem is it's painfully slow. The bottleneck is composed of two nested for loops. For each row of the data frame X, scan all rows of data frame Y, and does a series of checks inside an if-elif construct, to determine if the match is acceptable. The result is also supplied in the form of a data frame.

It's written in Python and mainly uses Pandas data frames. As far as I know it could be sped up only by a faster processor which I currently do not have access to. I tried parallel processing but the overhead between the loops was too big an it resulted in even logerlonger execution times (or my implementation was not good).

My question is, could I speed it up if I containerize itwith Docker and deploy it into a Kubernetes cluster with 2-3 nodes on my LAN?

Keep in mind that it is a linear application, I did not write any code to make it ready for parallel processing, and we are talking about a single process, not more processes initiated by more users. In this case, does Kubernetes know to distribute the workload (maybe the processing of the various iterations on the rows of data frame X ?) on the nodes, and then put it all together to deliver the answer?

Thank you!

I am fairly new to Docker and Kubernetes and I have a question for which I could not figure out the answer myself. I am working on an application that does string matching on data extracted from multiple sources but the problem is it's painfully slow. The bottleneck is composed of two nested for loops. For each row of the data frame X, scan all rows of data frame Y, and does a series of checks inside an if-elif construct, to determine if the match is acceptable. The result is also supplied in the form of a data frame.

It's written in Python and mainly uses Pandas data frames. As far as I know it could be sped up only by a faster processor which I currently do not have access to. I tried parallel processing but the overhead between the loops was too big an it resulted in even loger execution times (or my implementation was not good).

My question is, could I speed it up if I containerize it and deploy it in a Kubernetes cluster with 2-3 nodes on my LAN?

Keep in mind that it is a linear application, I did not write any code to make it ready for parallel processing, and we are talking about a single process, not more processes initiated by more users. In this case, does Kubernetes know to distribute the workload (maybe the processing of the various iterations on the rows of data frame X ?) on the nodes, and then put it all together to deliver the answer?

Thank you!

I am fairly new to Docker and Kubernetes and I have a question for which I could not figure out the answer myself. I am working on an application that does string matching on data extracted from multiple sources but the problem is it's painfully slow. The bottleneck is composed of two nested for loops. For each row of the data frame X, scan all rows of data frame Y, and does a series of checks inside an if-elif construct, to determine if the match is acceptable. The result is also supplied in the form of a data frame.

It's written in Python and mainly uses Pandas data frames. As far as I know it could be sped up only by a faster processor which I currently do not have access to. I tried parallel processing but the overhead between the loops was too big an it resulted in even longer execution times (or my implementation was not good).

My question is, could I speed it up if I containerize with Docker and deploy to a Kubernetes cluster with 2-3 nodes on my LAN?

Keep in mind that it is a linear application, I did not write any code to make it ready for parallel processing, and we are talking about a single process, not more processes initiated by more users. In this case, does Kubernetes know to distribute the workload (maybe the processing of the various iterations on the rows of data frame X ?) on the nodes, and then put it all together to deliver the answer?

Thank you!

Source Link
calin.bule
calin.bule
  • 3
  • 2

Can Kubernetes help with providing more processing power for the same request?

I am fairly new to Docker and Kubernetes and I have a question for which I could not figure out the answer myself. I am working on an application that does string matching on data extracted from multiple sources but the problem is it's painfully slow. The bottleneck is composed of two nested for loops. For each row of the data frame X, scan all rows of data frame Y, and does a series of checks inside an if-elif construct, to determine if the match is acceptable. The result is also supplied in the form of a data frame.

It's written in Python and mainly uses Pandas data frames. As far as I know it could be sped up only by a faster processor which I currently do not have access to. I tried parallel processing but the overhead between the loops was too big an it resulted in even loger execution times (or my implementation was not good).

My question is, could I speed it up if I containerize it and deploy it in a Kubernetes cluster with 2-3 nodes on my LAN?

Keep in mind that it is a linear application, I did not write any code to make it ready for parallel processing, and we are talking about a single process, not more processes initiated by more users. In this case, does Kubernetes know to distribute the workload (maybe the processing of the various iterations on the rows of data frame X ?) on the nodes, and then put it all together to deliver the answer?

Thank you!