We have a REST API Get request which downloads a file from the server. This file needs to be updated whenever there is a latest version. Currently this is done by deleting the existing file and moving the latest file as new one.

As the file is being held due to high traffic nature of service, deletion of file is not happening. This results in users getting older version of file.

How to re-design the system to update the file to its latest version while the file is being constantly accessed?

For deleting the file, we use the following

For moving the file, we use the following
File.Move(tempFilePath, filePath);

  • Are you using lock? If the file is not locked, then you should be able to delete (or rename/moving) it. A process that's already have the file open will still be able to read it, even after it gets deleted. That's how it works on Unix anyway – imel96 Dec 24 '20 at 5:33
  • No we are not using lock. Since the file is being constantly accessed replacing was not happening. – Anjo Dec 24 '20 at 6:02
  • Another possibility is the file replacement had happened but the server is still holding on to the old file's handle. If that's the case, the solution is to have a mechanism to reopen the file or for the server to reload – imel96 Dec 25 '20 at 21:31

Have the API point to a different file.

When an HTTP GET request arrives, the API won't just load (and send) a file from a hardcoded file path. Instead, if will:

  1. Check the directory for the available files.
  2. Find the latest version of the file (for instance, file-2.3.541.xml).
  3. Serve the latest version.

This way, pending requests continue to serve the old file, but new ones start serving the new one. Once you're sure there are no pending requests which rely on the old file, you can remove it. Or keep it for posterity, depending on the requirements.

Side note: there is no need to do two operations. File.Move has an additional parameter which allows to overwrite a file. Therefore, instead of a File.Delete followed by File.Move, you can simply do:

File.Move(tempFilePath, filePath, true);
  • Thanks for the answer. This will solve the current issue, but on reconsidering the design, we found out that the file is pretty much static and will change once in a month at max. So we decided to move the file to memory and serve requests instead of fetching it from file system. So there will be a memory cache loaded with this file (~150mb). Since we use a key to fetch value, how can we fix the above using key value model? – Anjo Dec 24 '20 at 6:00
  • 1
    @Anjo: not sure what's the question here. Caching is irrelevant to how files are named. You cache file-2.3.540.xml, and when file-2.3.541.xml becomes available, you cache it as well, and then wait until all requests to the previous version are finished in order to remove file-2.3.540.xml from cache. – Arseni Mourzenko Dec 24 '20 at 16:53

The problem with using a fixed file (i.e. the same file in the same path) is the source of your issue: the inability to update it without making it temporarily unavailable.

A better way to do this is to copy the new file next to the old one, wait for it to be completed, and then reconfigure your application to search for its file on the new path.

If your configuration is e.g. stored in a db or a refreshed config file, then you can adjust it without needing to take the application down, ensuring uptime for incoming requests.

You have some freedom here in how to specifically implement it, depending on your circumstances. Just don't try to hot-swap the same file with a live application - not unless you are okay with downtime while the file is being copied (which you are not, hence the question).

  • Thanks for the answer. We redesigned to keep 2 files one latest and 2nd one as the previous version. On re-considering the design, the file was pretty much static and hence decided to move to memory from file system. Now the issue is how to solve this issue using memory cache. Since cache uses key value model, only value(the file) changes when the latest version comes but the key remains same. – Anjo Dec 24 '20 at 6:06
  • @Anjo why the key is the same, the key should be the version. Once you have new file, you use new key which is the new version. As Flater said in his answer, you can read the latest version from a refreshed configuration file (db / server) and you are safe. – T.S Dec 25 '20 at 20:35
  • @T.S: The question mentions that the clients are supposed to get the latest file. So it stands to reason that there is also a "most recent" key that doesn't change with updates specifically because it always refers to the newest version. Even for a versioned file system, it's not uncommon to have a quick access "newest" option that doesn't require the user to specify a specific version. – Flater Dec 25 '20 at 22:18

Do not add version number in the name of the file (and its URL) which is to be fetched by REST API. Keep it always the same like my-soft.zip or mysoft.com. Keep versioned file names listed on "Other Versions" page.

  • Thanks for the answer. Could you please explain what is the issue in keeping the version number in name of the file? – Anjo Dec 24 '20 at 6:08
  • The issue with version number is that it keeps on changing. So, in order to keep filename and REST path static and quick to access, I have suggested to go without version number. For instance, you may see how wordpress.org has its latest package version download link and file name. It's wordpress.zip. – i50729 Dec 25 '20 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.