Is background long running thread a good choice for disk IO?

Question

I am building a storage engine software that allows concurrent data writing, now I have two different choices here:

Method 1. Background Long-Running Thread

Multiple user threads write to their own WriteBuffer
A background long-running IO thread iterate all write buffers and flush each write buffer one by one. (Need to obtain write buffer's mutex lock)
On each iteration, we can do a this_thread::yield() to prevent block other jobs too long.

Method 2. Use User Threads

Multiple user threads write to their own WriteBuffer
On each user thread writes, we use this user thread to iterate all write buffers and flush buffers. (also need to obtain buffers mutex)

Can anyone tell which one is a better choice and why? Or do you have any other suggestions ? Thanks.

What does "concurrent data writing mean"? Is it multiple users writing different files, located on the same HDD, at the same time? Or is it multiple users writing to the same file at the same time? Something else? — Bart van Ingen Schenau, Mar 1, 2021 at 8:23

Casey · Accepted Answer · 2021-03-01 01:47:23Z

Method 3: Producer-Consumer

A user requests a future write (with the corresponding data) by atomically appending the data to a thread-safe queue. (Producer)
The long-running background task pulls from this queue atomically a long as the queue is not empty. (Consumer)

If the queue is empty the long-running background task is put to sleep. Therefore, the disk is only hit when there is data to write.

This is used all the time for background logging to a file (particularly in games) and it has no noticeable affect on performance.

1 Answer 1