3

I'm designing an application that will be appending blobs to a file on disk (local filesystem) and I'm currently thinking of how to deal with consistency issues that could occur if:

  • The application suddenly crashes
  • The whole system stops, e.g. due to a power outage

The goal is that, when the file is later-on read, the application processing the blobs should be able to distinguish if a blob has been corrupted (and thus avoid processing it).

My current idea is to write the following on disk for each blob and flush after each one:

[Size of blob] (4 bytes) [CRC-32 hash of blob] (4 bytes, more to detect issues as files are aging over time) [actual blob bytes]

Here come the questions:

  • Does this guarantee that, should any of the above conditions occur, the file will contain either only valid data, or n valid blobs + some extra bytes where interpreting the first four as the size will easily indicate that there are not enough remaining bytes in the file for a proper blob (or the extra bytes are under 4, not enough to hold the proper size)?
  • Could a power-loss corrupt bytes that have been previously written to disk inside the file?
  • Could a power-loss corrupt the file such that it would appear much bigger than it should be (and thus contain various trash at the end)?
  • Can various filesystems lead to strange behaviors in this regard? The application will be cross-platform and I'm trying to avoid writing platform-specific code for this.

Some other considerations:

  • Blobs will be relatively small (around a few kB, < 100 kB)
  • Loosing a few blobs should a sudden stop occur is acceptable
  • When the application is restarted, it will create a new, empty file, not append to an already existing one
  • Only one thread of one process will be doing the appending
  • Reading the file before it is closed will not be allowed
  • Should a power-outage occur, a consistency check will be performed on the filesystem after rebooting.
  • This sounds like you're trying to write software to protect against hardware issues. While you should certainly be able to detect and cope with hardware failures (log the issues, provide the ability to failover to a redundant system, notify a system administrator, etc.) I would consider instead using protective systems and better IT policies. For example, UPS to protect against a power outage, RAID storage to protect against the failure of physical media, regular routine backup procedures to an off-site storage location, a redundant backup system for automatic failover, etc. – Ben Cottrell Apr 14 '17 at 11:03
  • Can't you try to use a logger system that will handle that for you ? – Walfrat Apr 14 '17 at 12:39
  • I could use SQLite for example which implements various features in this regard, but I think it's overkill for only storing blobs (as in event sourcing) and reading them in the order in which they where written. – D. Jurcau Apr 14 '17 at 12:48
  • 1
    Don't forget to account for the possibility that the underlying OS (or the underlying storage beneath that) may have write caching enabled. I would look to leverage an existing mature persistence layer rather than roll your own. All the various database engines will be able to store your data blobs using atomic transactions. – Thomas Carlisle Apr 14 '17 at 14:30
1

Unfortunately, your simple scheme does not protect you against disk failure or data corruption.

The weakness is in the unprotected size, which you need to read sequentially the file and find the next blob. So in case of a writing failure, on the size you can loose everything from the first bad size to the end of the file. This can happen in two variations:

  • creation of a new object: you write the data properly (say a blob of m bytes), and then eventually writing several other blobs. Imagine that the OS writes the size to the disk with an undetected corruption. When later you'll read again the file, you'll find out the wrong size n. There is a high probability that the CRC will highlight the inconsistency, but it will do on the n bytes that follow (although the blob was fully correct). Worse, the n bytes will be discarded as bad blob, and your code will from then on try to read the next blob at a wrong place (offset +n instead of offset + m).
  • file maintenance outside the application: for example, your blob file is copied from one server to a newer more performant one. If during the transfer a blob payload is corrupted, only this blob will be lost. However, if during the copy a size info gets corrupted, you'll loose all the subsequent blobs.

In a similar way, errors could also affect already written data. For example on an SSD, a hardware error could lead a bit to flip, a hardware defect could also affect disk cache memory (e.g. row hammer like effects). Some filesystems (or even hardware) try to rewrite an allocation unit, if it appears to be located on a defect hard drive sector), etc... But these are issues that affect most data structures, not only yours. One way to reduce them is to read the data you've written to cross-check its consistency.

0

Hrm....Could you simply originally write your blobs to some temp file, then change the file name when it's ready? That will significantly limit your fails on write and you can easily see which files have not made the change.

  • The blobs are constantly generated when various events occur, so it's hard to say when it's ready – D. Jurcau Apr 14 '17 at 12:02
  • But you know when you are done writing to it right? – unflores Apr 14 '17 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.