How do I recover from failed writes?

Question

I'm writing logic for a patchable game, and patching involves modifying multiple files.

type UpdateInfo {
    file string
    data []byte
    at   int64
}

func AtomicUpdate(ui <-chan UpdateInfo) error {
    for _, v := range ui {
        file, err := os.OpenFile(v.file, os.O_RDWR|os.O_CREATE, 0)
        if err != nil {
            return err
        }
        _, err = file.WriteAt(v.data, v.at)
        if err != nil {
            return err
        }
    }
}

ui represents a single atomic set of modifications that should be done in order to maintain a consistent state. The stakes of inconsistent state is users might have to redownload everything, which is pretty bad but isn't a catastrophe.

The question is then, what is a reasonable design to recover from a failed AtomicUpdate where state may be left inconsistent?

There is also a possibility of some interrupt (such as users tripping over their power) may happen during AtomicUpdate and doesn't even allow for normal error handling. Should this be a concern and how do I recover from that?

Answer 1

The safest way to update a group of related files is to use an algorithm like this

1. Create a copy of each involved file (or of the whole tree of files)
2. Apply the patches to the copies of the files
3. After all patches have been applied, replace the original files with the (modified) copies. Use a filesystem action to change the name of a file/folder for this.
4. If an error occurred during applying the patches, delete the copy.

This way, only during the last step is there any risk that a patch gets applied partially, even if there is a power outage. That last step is as fast as possible, and might actually be atomic if you make the copy on a tree level.

Answer 2

Did this 25 plus years ago applying patches to military battle field equipment over the air - failure was not an option. The system we designed was more elaborate for various reasons, but was essentially designed around the idea of a few steps.

1) Create a "in progress" file 'started'. 2) Backup files. 3) Update "in progress" file 'backup complete'. 4) Patch files 5) remove "in progress" file.

On startup, if the "In progress" file exists, examine it to find out where in the process it failed again (or restore and fail). All sorts of options exist for error detection, roll forward, roll back etc. System probably should involve hashes and is needed for security, signatures.

Unfortunately its one of the programming problems that is not particularly hard to get right, but is very easy to get wrong.