I have a class that is responsible for writing formatted binary objects to a file on a network drive with a 100Mbps bandwidth.
Each time an object is created (via lets call it
w.addObj()) it formats it in its binary representation and written to file. This operation may take place millions of times, and calling
write() that many times does not make use of the network speed. In fact, writing 76e6 objects takes about 10 minutes (total file size will be ~2GB). To overcome I decided to provide an internal buffer to the class that will store the binary objects until some size limit is reached and then flush to file. Currently I set this “buffer” to 100MB but that number is arbitrarily determined.
Ideally a solution that improves writing speed (as sequentially writing is slow), that is memory efficient and maximizes the available write speed of the network. Is it possible to determine such an ideal chunk size (ie what I currently set to 100MB) that achieves this? Or does it simply result from testing different buffer/chunk sizes?