Here is the question.
On a 2.2 GHz processor with 6 physical threads and 6 hyperthreads, I see performance on the order of 3-10s for a particular job involving OpenCV.
I do not specify that the process should use many threads on this local machine, but all threads become active during use.
When this task is submitted to a Kubernetes POD with Xeon Platinum 2.5 GHz processor w/ all the bells and whistles, it runs at times anywhere from 3-10s to 100s of seconds.
I have the following hypotheses:
- we build on a non-Xeon box with CI/CD, so I need to manually compile OpenCV for the Xeon w/SSE and compiler optimization flags
- libomp gets leveraged for SVD calculations (and other things), which uses many threads no matter what...so I need to increase the thread count of the PODs (alternatively: my laptop is just load balancing the cores to be thermally efficient, and it is still running on a single "thread")
- Hyperthreads and Physical threads are split across separate VMs in the physical box in the datacenter and algorithms leveraging cache-optimizations are negatively impacted
- The onboard BIOS of the processor in the data center is throttling the GHz of the processor because the thermal load is too high (the data center is running hot)
In the last case,
The onboard BIOS of the processor in the data center is throttling the GHz of the processor because the thermal load is too high (the data center is running hot)
My question is: is this a reasonable hypothesis?
What could possibly be the cause of this slowdown?