3

How to implement a multi-threaded program in which each thread needs to write/output some data (in any order of course) but without the locking overhead that is involved with files or a particular data structure.

Is each thread writing to a separate file a feasible option? (wrt to locking overhead)

Can we use System.out.print or BufferWriter instead of files? Or is there any other elegant solution?

  • Another thing to note: the data written by each thread is less than 100 bytes, however the number of threads can be close to a hundred. – Outflanker May 3 '15 at 2:47
  • LinkedTransferQueue documentation – rwong May 3 '15 at 3:04
  • What exactly do you mean by "locking overhead that is involved with files". Maybe you mean blocking characteristic of file operations? In that case, producer-consumer pattern could help. – qbd May 3 '15 at 8:08
1

If you have multiple threads generating data independently from each other, writing to separate files may be indeed a good option. There will still be some locking under the hood since the Operating System needs to keep the filesystem consistent, but it should be better than writing to a single file with manual locking or writing to standard output (which has internal synchronization). Similarily if you have access to some online storage which you can access via the network, you could send the data independently from each thread.

But there is a big "but". If you write out a lot of data (more than several MB/s, depending on machine specs), your file system may not be able to keep up. In such a situation, the bottleneck will be your file system and not locks in the app. If this happens, you might actually be better off with a single file and locking since concurrent writes to multiple locations may put more load on the filesystem and disk than writing at a single spot. So it all depends and any number of concurrent writes between one and many may be optimal. Your best bet would be to make the number of concurrent writes parametrizable and then to compare the final performance of the application for several different values.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.