Standalone WAL (Write Ahead Log) implementation

Question

I'm exploring how I might build a Write Ahead Log. My Use case is for operations like:

• Log Messages (INFO, DEBUG, etc)
• File Uploads

Where I would write the some information representing the Log Message or a File Upload to the WAL, then attempt flush that message to it's final destination. Effectively what I am after is a persistent queue. I figured that a WAL fits here and should be easily implemented without additional dependencies, maybe I'm wrong so feedback is appreciated, as my application should be able to continue / restore in the case of a power failure or some other crash.

My device in this case is a Kiosk, so it will basically be alone in the world by itself just humming along until some bad thing happens: someone accidentally unplugs it for example and it eventually gets powered back on. I figured the WAL fits the scenario for recovery on application start up.

To that end I am looking at System.IO.MemoryMappedFiles.MemoryMappedFile as a potential implementation solution.

What I have sketched out so far is running 2 mmap's handles.

1. the WAL
2. an index on the WAL

The index's header would store the current HEAD and the current COMMIT. I'm using COMMIT here to mean "yes, everything up to this point has been dealt with successfully"

Each index point after the HEAD and COMMIT would store the location of WAL data in the WAL file.

so the index would look something like (each index entry representing a 4 byte int):

Head        Commit      Index 0     Index 1
v           v           v           v
00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 58

Log Entry 0 begins at byte 0 in the WAL file

Log Entry 1 begins at byte 88 in the WAL file

Thus to read an entry from the WAL would be a matter of reading index N to get the start of the range and then reading index N + 1 to get the end (aka the next start).

Behind the scenes I'd have some interval to run Checkpoints and clean up the Index and the WAL. I'm getting ahead of myself there as my main purpose with this question is to get feedback on weather or not this is even a good idea and if so, if using a MemoryMappedFile for the WAL and the WAL index is the best way to handle it.

In a case like capturing log messages, there could be quite a bit of writing happening in a short about of time.

Answer 1

"Best" is highly dependent on usage scenarios and the criteria for notion of "best".

If you are writing a log that is read only during restart/recovery then I would think that ordinary buffered writes to an append only file would be best.

For writing an append only log that itself is the database, and thus might have older update records frequently being read, the memory mapped concept might be better.

Beyond that there are some other things to say:

On the architecture of the update records in the log: I would probably integrate the content of the updates with the index information, if they are only processed serially such as in recovery. Update records having a header that refers to the previous update record, the content, and then a trailer that has a hash or checksum of some sort for later validation of the update, along with a reference to the beginning of the update record. On the other hand if some consumer uses random reads of updates then an index might help.

I believe that, generally speaking, an append only usage of files would be friendlier to a SSDs than updating the first bytes of the file along with each append, as this would result in fewer discontiguous writes.

On using memory mapped files, if you are appending to a memory mapped file, you will need to have mapped into memory using a size larger than the file presently is: this for holding the appended stuff. When you reach the end of the mapping size, you have to remap the file (larger), as windows does not allow extending the mapping as you go. So, what I do is map the file with a large amount of slack, and later truncate the file to the proper length. Of course, I am still prepared to remap the file if I run out of slack.

Windows maps files in pages, and it does not track the actual EOF (the act of mapping the file with slack changes the file's EOF) so maintaining the EOF is the application's responsibility: one can fix the EOF on graceful application shutdown after unmapping the file. However, if the app or system crashes (e.g. power lost), the file will probably have extra pages well past the intended EOF. So what I do is look for the last update record searching backward from the file's EOF (here's where hashes help with validation).