Note this is a general, conceptual question about performance optimization. motivated by the following real-world case.
I have a file on a Windows network drive that has a 100Mbps limt; it is a binary file and is 165MB.
My local machine has software on it specifically designed to manipulate this file format, and when opened in that software takes less than a second to display all the information. When monitoring the Task Manager during this split second, the process for the software shows:
- 13% Network (@ 26.4 Mbps briefly)
- 08% CPU (@ 1.2% briefly)
Since the format is known, I wrote a Python script to parse it and the fastest that I can do while using the
struct module is roughly 15-17 seconds. During this time, CPU usage for the Python process doesn't change, but Network usage reaches 94% (@ 82 Mbps on average).
What could a software be doing that its able to fully read the file so fast, yet I'm maximizing the network bandwidth and it takes me much longer?