0

Here is the question.

On a 2.2 GHz processor with 6 physical threads and 6 hyperthreads, I see performance on the order of 3-10s for a particular job involving OpenCV.

I do not specify that the process should use many threads on this local machine, but all threads become active during use.

When this task is submitted to a Kubernetes POD with Xeon Platinum 2.5 GHz processor w/ all the bells and whistles, it runs at times anywhere from 3-10s to 100s of seconds.

I have the following hypotheses:

  • we build on a non-Xeon box with CI/CD, so I need to manually compile OpenCV for the Xeon w/SSE and compiler optimization flags
  • libomp gets leveraged for SVD calculations (and other things), which uses many threads no matter what...so I need to increase the thread count of the PODs (alternatively: my laptop is just load balancing the cores to be thermally efficient, and it is still running on a single "thread")
  • Hyperthreads and Physical threads are split across separate VMs in the physical box in the datacenter and algorithms leveraging cache-optimizations are negatively impacted
  • The onboard BIOS of the processor in the data center is throttling the GHz of the processor because the thermal load is too high (the data center is running hot)

In the last case,

The onboard BIOS of the processor in the data center is throttling the GHz of the processor because the thermal load is too high (the data center is running hot)

My question is: is this a reasonable hypothesis?

What could possibly be the cause of this slowdown?

Improve this question
7
  • AFAIK, datacenters use machines with AMD and Intel CPUs just as anyone else, and for both types throttling as part of heat and power management has been pretty standard for years. What I cannot tell you is if this can explain the performance difference you observed (I would expect this to be one factor among many others).
    – Doc Brown
    Jan 12, 2022 at 17:11
  • @DocBrown I am behind the scenes and lack direct control of this. My ability to influence runtime is related to my ability to come up with good ideas for someone else to try, and manage the dockerfile. From the dockerfile, I can compile optimal binaries w/the variety of build systems, so this is an option. But I do not see a clear prioritization of the things to try. Perhaps I need to rewrite the question (and will do so), but do you see any outstanding factors that could, at least in some way, explain this level of slowdown?
    – Chris
    Jan 12, 2022 at 18:34
  • I am pretty sure don't know enough about your system to make any kind of meaningful guess. I would recommend you add some logging (with timestamps!), so you find out where in the whole process the unexpected performance loss occurs. For example, is it really the job itself which is slowed down, or is it the start-up time, or is it a particular step within the job? Instead of blindly guessing around, start to collect more data first.
    – Doc Brown
    Jan 12, 2022 at 20:36
  • A significant presumption here is that the task does the same amount of work every time it is run. If this is true it should be stated and justified in the question.
    – candied_orange
    Jan 12, 2022 at 22:48
  • 2
    Instrument first, Collect Data, Look for culprits. I heard docker, have you controlled for other containers running, and for mesh routing overheads?
    – Kain0_0
    Jan 13, 2022 at 6:31

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy