7

We have a REST API Get request which downloads a file from the server. This file needs to be updated whenever there is a latest version. Currently this is done by deleting the existing file and moving the latest file as new one.

As the file is being held due to high traffic nature of service, deletion of file is not happening. This results in users getting older version of file.

How to re-design the system to update the file to its latest version while the file is being constantly accessed?

For deleting the file, we use the following
File.Delete(filePath)

For moving the file, we use the following
File.Move(tempFilePath, filePath);

Improve this question
3
  • Are you using lock? If the file is not locked, then you should be able to delete (or rename/moving) it. A process that's already have the file open will still be able to read it, even after it gets deleted. That's how it works on Unix anyway
    – imel96
    Dec 24, 2020 at 5:33
  • No we are not using lock. Since the file is being constantly accessed replacing was not happening.
    – Anjo
    Dec 24, 2020 at 6:02
  • Another possibility is the file replacement had happened but the server is still holding on to the old file's handle. If that's the case, the solution is to have a mechanism to reopen the file or for the server to reload
    – imel96
    Dec 25, 2020 at 21:31

3 Answers 3

Reset to default
11

Have the API point to a different file.

When an HTTP GET request arrives, the API won't just load (and send) a file from a hardcoded file path. Instead, if will:

  1. Check the directory for the available files.
  2. Find the latest version of the file (for instance, file-2.3.541.xml).
  3. Serve the latest version.

This way, pending requests continue to serve the old file, but new ones start serving the new one. Once you're sure there are no pending requests which rely on the old file, you can remove it. Or keep it for posterity, depending on the requirements.

Side note: there is no need to do two operations. File.Move has an additional parameter which allows to overwrite a file. Therefore, instead of a File.Delete followed by File.Move, you can simply do:

File.Move(tempFilePath, filePath, true);
Improve this answer
2
  • Thanks for the answer. This will solve the current issue, but on reconsidering the design, we found out that the file is pretty much static and will change once in a month at max. So we decided to move the file to memory and serve requests instead of fetching it from file system. So there will be a memory cache loaded with this file (~150mb). Since we use a key to fetch value, how can we fix the above using key value model?
    – Anjo
    Dec 24, 2020 at 6:00
  • 1
    @Anjo: not sure what's the question here. Caching is irrelevant to how files are named. You cache file-2.3.540.xml, and when file-2.3.541.xml becomes available, you cache it as well, and then wait until all requests to the previous version are finished in order to remove file-2.3.540.xml from cache.
    – Arseni Mourzenko
    Dec 24, 2020 at 16:53
3

The problem with using a fixed file (i.e. the same file in the same path) is the source of your issue: the inability to update it without making it temporarily unavailable.

A better way to do this is to copy the new file next to the old one, wait for it to be completed, and then reconfigure your application to search for its file on the new path.

If your configuration is e.g. stored in a db or a refreshed config file, then you can adjust it without needing to take the application down, ensuring uptime for incoming requests.

You have some freedom here in how to specifically implement it, depending on your circumstances. Just don't try to hot-swap the same file with a live application - not unless you are okay with downtime while the file is being copied (which you are not, hence the question).

Improve this answer
3
  • Thanks for the answer. We redesigned to keep 2 files one latest and 2nd one as the previous version. On re-considering the design, the file was pretty much static and hence decided to move to memory from file system. Now the issue is how to solve this issue using memory cache. Since cache uses key value model, only value(the file) changes when the latest version comes but the key remains same.
    – Anjo
    Dec 24, 2020 at 6:06
  • @Anjo why the key is the same, the key should be the version. Once you have new file, you use new key which is the new version. As Flater said in his answer, you can read the latest version from a refreshed configuration file (db / server) and you are safe.
    – T.S
    Dec 25, 2020 at 20:35
  • @T.S: The question mentions that the clients are supposed to get the latest file. So it stands to reason that there is also a "most recent" key that doesn't change with updates specifically because it always refers to the newest version. Even for a versioned file system, it's not uncommon to have a quick access "newest" option that doesn't require the user to specify a specific version.
    – Flater
    Dec 25, 2020 at 22:18
0

Do not add version number in the name of the file (and its URL) which is to be fetched by REST API. Keep it always the same like my-soft.zip or mysoft.com. Keep versioned file names listed on "Other Versions" page.

Improve this answer
2
  • Thanks for the answer. Could you please explain what is the issue in keeping the version number in name of the file?
    – Anjo
    Dec 24, 2020 at 6:08
  • The issue with version number is that it keeps on changing. So, in order to keep filename and REST path static and quick to access, I have suggested to go without version number. For instance, you may see how wordpress.org has its latest package version download link and file name. It's wordpress.zip.
    – Rishi Raj
    Dec 25, 2020 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.