0

I have a class that is responsible for writing formatted binary objects to a file on a network drive with a 100Mbps bandwidth.

Each time an object is created (via lets call it w.addObj()) it formats it in its binary representation and written to file. This operation may take place millions of times, and calling write() that many times does not make use of the network speed. In fact, writing 76e6 objects takes about 10 minutes (total file size will be ~2GB). To overcome I decided to provide an internal buffer to the class that will store the binary objects until some size limit is reached and then flush to file. Currently I set this “buffer” to 100MB but that number is arbitrarily determined.

Ideally a solution that improves writing speed (as sequentially writing is slow), that is memory efficient and maximizes the available write speed of the network. Is it possible to determine such an ideal chunk size (ie what I currently set to 100MB) that achieves this? Or does it simply result from testing different buffer/chunk sizes?

Improve this question
6
  • Is this a one off for your current system or are you writing code that will need to discover this ideal chunk size once deployed? – candied_orange Sep 30 '20 at 18:38
  • 2
    Depending precisely how you write to this file, the language and libraries you use may already be buffering the actual writes when you 'flush' to file. Testing is the best answer. For example, start at 8K and double it until you see no improvement. And recognize that the answer in your production environment may be different from the test environment. – joshp Sep 30 '20 at 19:01
  • @candied_orange the interface will be part of a library that will be deployed for any user, so it should not be specific to my particular network. I’ve quoted my network here because that’s where I use it, but sequentially writing only used 9Mbps on average when I would like to maximize it. – datta Sep 30 '20 at 19:50
  • @joshp the language being used is Python (CPython) which does buffer underneath, but setting the buffering size is still the question at hand: how to determine the size rather than defaulting not the default page sizing (per the docs)? – datta Sep 30 '20 at 19:52
  • 3
    @datta Ok, then you have my advice. Testing is the best way to find out how important the. buffer size is and. what. is the optimal size, – joshp Oct 1 '20 at 4:01
2

There are three factors at play:

  • performing fewer large request tends to have higher throughput
  • waiting for a buffer to fill up increases latency
  • buffers require RAM

This is a multi-objective optimization problem: we want to maximize throughput and minimize latency, subject to the constraint that there has to be enough free memory.

However, that is a multi-objective problem, which means that there isn't a single best solution. Any buffer size could be the best throughput:latency tradeoff.

Because you do not mention latency requirements, this could indicate that latency is not important and the biggest feasible buffer size would be optimal. What is feasible depends drastically on the system you're deploying on. A Raspi with 2GB RAM is entirely different from a server with half a TB of memory. And there may be other applications running.

When writing a library, you should just keep the buffer size configurable and let the application decide. You could perhaps set a sane default that will lead to reasonable performance without impacting users, so perhaps 50MB for a permanently allocated buffer when deployed on a laptop, or larger buffers if it can be freed afterwards (potentially tricky in Python).

I am slightly confused though because a file.write() call in Python doesn't necessarily perform the write immediately. Both the Python file and the operating system may have buffers of their own, typically in the range of a few KB. E.g. Python's io.BufferedWriter and open() function default to the system's file block size, typically 4KB or 8KB (can be overridden). And when an actual write is performed, this should just block until the file system driver acknowledges the write, not necessarily until the write is actually durable. But this may depend on how your network file system is mounted.

Improve this answer
2
  • I though somewhere in the 10-50MB range would be ideal, and I’ll probably keep it at 10MB since the network the library will be used on the most is limited to 100Mbps which is 12.5MB/s so that should limit the number of write() calls while maximizing the available bandwidth. As to the confusion regarding Pythons implementation of file.write(), the binary objects were being written to file so frequently (because they were filling the buffer so fast) that it took 10 minutes to write; if I stored in a buffer and wrote in 100MB chunks it takes 2.7 minutes. – datta Oct 1 '20 at 16:17
  • Therefore, I wanted to see if there was an ideal way to determine how large my internal buffers should be. The reason I use internal buffers rather than setting the buffer parameter of an open file is due to an implementation detail where I needed to check the current size for something else; so when the 100MB was reached I did something else in addition to writing to file. The systems my team use are insane compared to machines it will be deployed to, so I wanted something pragmatic. Thus, 10MB will improve the speed while also taking less than a second to run and appear seem less to users. – datta Oct 1 '20 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.