2

Note this is a general, conceptual question about performance optimization. motivated by the following real-world case.

I have a file on a Windows network drive that has a 100Mbps limt; it is a binary file and is 165MB.

My local machine has software on it specifically designed to manipulate this file format, and when opened in that software takes less than a second to display all the information. When monitoring the Task Manager during this split second, the process for the software shows:

  • 13% Network (@ 26.4 Mbps briefly)
  • 08% CPU (@ 1.2% briefly)

Since the format is known, I wrote a Python script to parse it and the fastest that I can do while using the struct module is roughly 15-17 seconds. During this time, CPU usage for the Python process doesn't change, but Network usage reaches 94% (@ 82 Mbps on average).

What could a software be doing that its able to fully read the file so fast, yet I'm maximizing the network bandwidth and it takes me much longer?

Improve this question
20
  • 6
    Does it really display all the information? Or is it just very fast as displaying the information it needs? Unless the program is cheating and caching things on your machine, the only way I can see it doing what it does is reading only a subset of the file. – Sebastian Redl Feb 28 '20 at 21:04
  • 1
    Assuming those rectangles use doubles for their points, that's only 1.1 to 2MB of data or so for just the rectangles. There is a lot of extra information in that extra 163MB that isn't being displayed right away. – whatsisname Feb 28 '20 at 21:20
  • 1
    @whatsisname Correct. So for 36000 shapes thats (5 * 16 * 36000) or 2.88MB. Perhaps the software is just reading the points and skipping everything else? I don't see how it would do that because some of that information is critical. – datta Feb 28 '20 at 21:25
  • 1
    I don't know a ton about windows networking, but maybe its using compression? – GrandmasterB Feb 28 '20 at 21:43
  • 1
    @datta: is it an arbitrary shapefile, or is it a specific one which may be optimized by holding the same geometry information in different level of details? Using different LODs is a standard optimization technique to support different zoom levels very quickly. – Doc Brown Feb 29 '20 at 9:42
9

A network of 100 Mbps (mega bits per seconds) conveys 12,5 MB per seconds, including payload and protocol overhead. A file of 165 MB needs at least 13,5 seconds (In fact, it would require slightly more: data has to be packed into IP packets of maximum 64KB, each having an additional 40 byte header).

So it is strictly impossible that the application can read the full data in less than a seconds.

There are several possibilities however to achieve the behaviour that you describe:

  • if the file format would be extremely redundant and the server side could compress it to only 7% of it’s original size. This is very improbable.
  • if the app would have read the data before (e.g. cached from previous session? or background loading?)
  • if the app reads only a small part of the file (less than 12,5 MB), display immediately the partial content to the screen, and while you wonder how fast it is and start reading the content, starts reading the rest if the file in background.

The last hypothesis is by far the most probable.

Improve this answer
1
  • 2
    That’s what would happen with a 165MB video file. All the information about the file takes very little space. Most of the 165MB is only read as you play the video. – gnasher729 Feb 29 '20 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.