0

I am building a storage engine software that allows concurrent data writing, now I have two different choices here:

Method 1. Background Long-Running Thread

  • Multiple user threads write to their own WriteBuffer
  • A background long-running IO thread iterate all write buffers and flush each write buffer one by one. (Need to obtain write buffer's mutex lock)
  • On each iteration, we can do a this_thread::yield() to prevent block other jobs too long.

Method 2. Use User Threads

  • Multiple user threads write to their own WriteBuffer
  • On each user thread writes, we use this user thread to iterate all write buffers and flush buffers. (also need to obtain buffers mutex)

Can anyone tell which one is a better choice and why? Or do you have any other suggestions ? Thanks.

1
  • 1
    What does "concurrent data writing mean"? Is it multiple users writing different files, located on the same HDD, at the same time? Or is it multiple users writing to the same file at the same time? Something else? Mar 1, 2021 at 8:23

1 Answer 1

1

Method 3: Producer-Consumer

  • A user requests a future write (with the corresponding data) by atomically appending the data to a thread-safe queue. (Producer)
  • The long-running background task pulls from this queue atomically a long as the queue is not empty. (Consumer)

If the queue is empty the long-running background task is put to sleep. Therefore, the disk is only hit when there is data to write.

This is used all the time for background logging to a file (particularly in games) and it has no noticeable affect on performance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.