0

I am writing a Windows Service using the Windows Background Worker Service template. Some data that I track needs to persist between application instances (i.e. after shutdown and restart).

Initially, I saved this data into the app.config.json file. However, this involved clobbering the file. Normally, this wouldn't be a problem. However, I have had an instance where the machine didn't shutdown properly (either because the user forced a shutdown, or cut the power) and the file contents where deleted but never replaced. This resulted in the application losing all of it's configuration, breaking it in the process.

Is there a better/more reliable way of storing persistent data? I'm thinking about an embedded database, but I believe that may be overkill for my situation--I only have a few data points. Any thoughts?

EDIT: I agree with @Euphoric that saving the data to a separate file, and not modifying the config is a better option.

However, it doesn't address the fundamental issue. Namely, is there a way to save that data in a fail-safe way? Obviously, saving it to a file will always carry risk. But is there a way to mitigate data-loss even in exceptional circumstances?

My only idea is to update the data when it is generated, rather than on shutdown. However, this carries a significant performance cost from open/closing the file.

4
  • 1
    Create a new file?
    – Euphoric
    Jun 7, 2023 at 13:56
  • @Euphoric I agree, but it still leaves the problem of the data not being saved to the file on shutdown. I can't revert to the old data, since it's dynamic.
    – PatJon6
    Jun 7, 2023 at 15:23
  • 2
    But is there a way to mitigate data-loss even in exceptional circumstances? saving to disk periodically or after every change instead of only once and wishing for the best. And no, you can't make those frequent writes "safer" with another technique that will likely need the same technique to make it safer, and so on and so forth. It's a matter of probabilities. N attempts to write in disk vs 1.
    – Laiv
    Jun 7, 2023 at 15:29
  • Your program could crash before, during or after writing the new state. A crash before the write means new state is not written, you can’t get round that.
    – gnasher729
    Jun 8, 2023 at 20:18

4 Answers 4

2

I suggest something similar to double buffering.

Maintain two versions of the file, app.config.jsoncheck.1 and app.config.jsoncheck.2. Store them in such a manner that the last 64 (hex) characters in the file are the SHA-256 checksum of everything except the last 64 characters. Do the writing periodically, not just on application shutdown.

When the application starts, check which file is newer. Check if the checksum is valid. If so, use that file. If not, use the other file. Presumably on unclean shutdown both files aren't corrupted, at least if the write to disk happens at intervals longer than few minutes.

If you can't rely on operating system file timestamps, then obviously you could store the version number in the file too, every time the file becomes newer incrementing it by 1. Then you know merely from the file contents which one is newer.

1

As @Euphoric hints at, writing a new file (and renaming it) is the standard way of doing this. So, the steps would be:

  • write the configuration data to app.config.json.new
  • rename the old file from app.config.json to app.config.json.old
  • rename the new file from app.config.json.new to app.config.json
  • optionally delete app.config.json.old (unless you want to keep it in case you need to revert)

There is a small window between the two rename operations in which there is no file app.config.json, and if your shutdown happens in that window, your startup needs to handle that. Thus the startup procedure should check for app.config.json, app.config.json.new, and maybe app.config.json.old, and use the first file found.

3
  • If the file gets corrupted, duplicating the file during bootstrapping won't make any difference.
    – Laiv
    Jun 7, 2023 at 14:18
  • I meant more of a creating a separate file for storing runtime configuration. Applicatoin shouldn't be changing app.config.json
    – Euphoric
    Jun 7, 2023 at 14:19
  • 1
    MacOS has an atomic “exchange file” operation for this purpose. So write to copy, exchange files, delete copy. Maybe you have something equivalent available.
    – gnasher729
    Jun 7, 2023 at 17:53
1

Here is a typical pseudo-algorithm for writing changes to the filesystem frequently, but not too frequently.

  1. Suppose the user is "fiddling" with the changes.
  2. Each time any change is made, a C# countdown timer is started, with a limit of N seconds.
  3. Any time a change is made, the same timer will be reset to its original state of N seconds.
  4. If the timer is ever triggered, that necessarily implies that whole N seconds have elapsed when the user did not "fiddle" with the changes. That could mean:
    1. The user is satisfied with the changes, or
    2. The user is taking a short break.
  5. In either case, it is prudent to save the changes to the filesystem. Meanwhile the timer should be put in the inactive state, unless and until the user makes another change.

The choice of N depends on how aggressive you need to be in order not to lose any changes, versus how much (potentially redundant) disk activity you can tolerate.

Regarding the clobbering issue, generally speaking it is necessary to study the C# (and also the underlying Win32 API) file system API and understand its features and options and guarantees. For example, the flag FILE_FLAG_NO_BUFFERING is used to ensure every read and write operation goes directly to the physical file system. https://learn.microsoft.com/en-us/windows/win32/fileio/file-caching

That said, one must also understand the fundamental limits of these approaches, and understand that a sufficiently determined user can always defeat any clever scheme. For example, a user can somehow configure the software system so that the file to be written is somehow located on a removable drive (e.g. a USB drive). To defeat the writing of the file, the user simply unplugs the USB drive at the most convenient time. Even worse, there are plenty software behavior monitoring tools which can be used to detect exactly what Win32 API options are specified when certain files are opened, and these too can be used to the adversarial user's advantage.

Try harder, but don't try too hard.

1

I think that if you wait for the instance to shut down/restart gracefully to save the configuration, then you are probably late.

In line with @Euphoric's comment, a reliable failover would be a never-changing file with the default configuration (default-app.config.json). This file is meant to be the "Plan B" in case app.config.json is empty, not accessible or corrupted. In any of these cases, the service makes a copy of default.app.config.json named app.config.json and loads this new file.

To prevent losing the configuration at the last moment due to unexpected shutdowns, a solution could be frequent writes to disk. As logs, you can make and rotate copies. For example

  1. Write the configuration into a .tmp file
  2. Move app.config.json to app.config.bck
  3. Move .tmp file to app.config.json

Notice I write only once; the rest is done moving files. In Linux (hopefully, also in Windows) moving files is faster and safer than copying.

If the config file is critical, and you can't restore it with default.app.config.json, this strategy falls short. A better one will involve file replication in multiple storages. The replication, better managed at operating system or hardware level.

3
  • I'm unclear as to what you're suggesting. When do I write to the .tmp file? At shutdown?
    – PatJon6
    Jun 7, 2023 at 15:33
  • the idea would be frequent writes to disk in every cycle, you repeat the 3 steps. Once you have this, you can tune it up a bit. For example, writing frequently only if the configuration changed. You can track changes in memory, for example by storing hashes to make check Sums.
    – Laiv
    Jun 7, 2023 at 15:34
  • Moving (renaming) files is not only a faster operation, it is also more likely to be an atomic operation, making it easier to recover if the computer lost power during the two move operations. Jun 8, 2023 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.