53

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was looking, I found it in other standard applications as well.

These programs first write to the temp folder, and then copy it to the intended destination.

Example: 7zip first extracts to the temp folder, and then copies the extracted data to the location that you had asked it to extract the data to.

I see several problems with this approach:

  1. The temp folder might not have enough space, while the intended location might have that much space.

  2. If it is a large file, it can take a non-negligible amount of time for the copy operation.

I thought about it a lot, but I couldn't see one single positive point to doing this. Am I missing something, or is there a real benefit to doing this?

Improve this question
4
  • If the application requires it, do it. Otherwise, do what makes sense. Your brain is telling you that its not needed in this case. I'm in the camp of doing the least amount of work possible as you can tell by my answer so I recommend doing the least amount of work possible for the app.
    – King Friday
    Commented Jun 16, 2012 at 19:40
  • 1
    Are you sure it is "copying" the file and not "moving" it? Big difference.
    – frankc
    Commented Jun 29, 2012 at 20:00
  • From blogs.rsa.com/…, for malwares, the advantage is 1) Guaranteed writability avoids "red error" detection before you have even fully downloaded the virus onto the user's computer 2) TEMP disks are usually faster (RAMDisk) reducing time for detection while the virus is still in downloading stage (once the virus is fully formed and managed to execute for 0.05 seconds, the game is over, so it's a race of time)
    – Pacerier
    Commented Jul 29, 2017 at 13:21
  • 3) While you're still downloading or running, if an antivirus smashed into you (itself without knowing, sometimes) and left your bones lying around, you can still count on an x% chance that the OS or user would accidentally help you delete your dead bones, while they are bulk clearing stuff from TEMP folder, this helps to reduce detection probability.
    – Pacerier
    Commented Jul 29, 2017 at 13:26

3 Answers 3

Reset to default
101

A few reasons I can think of:

  • On most platforms, file moves are atomic, but file writes are not (especially if you can't write all the data in one go). So if you have the typical producer/consumer pattern (one process produces files, the other watches a directory and picks up everything it finds), writing to a temp folder first and only then moving to the real location means the consumer can never see an unfinished file.
  • If the process that writes the file dies halfway through, you have a broken file on your disk. If it's in a real location, you have to take care of cleaning it up yourself, but if it's in a temp location, the OS will take care of it.
  • If the file happens to be created while a backup job is running, the job may pick up an incomplete file; temp directories are generally excluded from backups, so the file will only be included once moved to the final destination.
  • The temp directory may be on a fast-but-volatile filesystem (e.g. a ramdisk), which can be beneficial for things like downloading several chunks of the same file in parallel, or doing in-place processing on the file with lots of seeks. Also, temp directories tend to cause more fragmentation than directories with less frequent reads, writes, and deletes, and keeping the temp directory on a separate partition can help keep fragmentation of the other partitions down.

TL;DR - it mostly boils down to atomicity, that is, you want to make it so that (at the final location) the file is either complete or not there at all at any given time.

Improve this answer
8
  • 13
    If the temp directory is on a separate partition, you lose the atomicity.
    – yfeldblum
    Commented Jun 16, 2012 at 16:43
  • 17
    Some programs will extract/copy to the intended directory, but they will use a temporary file extension (e.g. .tmp) and rename it when complete.
    – Dan Diplo
    Commented Jun 16, 2012 at 16:43
  • 5
    Sometime writing to a temp file is also useful when several instances of your program could try to upload the same file at the same time, and you cannot rely on the file system to provide the right locking level (like poorly-defined cloud storage API). Using temp file will make sure the resulting file won't end up with a mix of data from both uploads. Of course this is just another instance of atomicity.
    – Krzysztof Kozielczyk
    Commented Jun 16, 2012 at 19:18
  • 2
    Dan Diplo's case is useful when updating an existing file. You don't want to replace the old one until the new one is successfully written, lest the write of the new one fail or something else read the new one before it's fully written.
    – RalphChapin
    Commented Jun 18, 2012 at 14:29
  • 1
    There is one exception to this rule and that is when the process writing to the file is only appending to the end of it, and then closing it after every append operation. This sort of procedure is highly efficient, and is often used when writing to potentially large log files.
    – dodgy_coder
    Commented Jun 22, 2012 at 0:34
15

This seems to be an issue in Windows, more specifically related to how the drag-drop is managed.

The developers of the WINSCP client have developed their own shell extension, which overrides this drag-drop behavior and allows dropping the file to the right folder immediately. They explain the trick in their documentation and, more interesting, what is the problem and how they solved it.

Here is the interesting part:

Windows drag&drop mechanics does not allow source application of drag&drop operation to find out easily, where the files are dropped. It is up to target application (Windows Explorer usually) to transfer files to destination. It is rather reasonable, because source application can hardly transfer files to all possible destinations. Keep in mind that you can drop files not only to a directory, but even to ZIP file (or any other archive), remote directory (via FTP, SFTP, SCP, …), trash, …

Obviously even Windows Explorer (or any other target application, like WinZip) cannot download files from any possible source (particularly it does not know SFTP/SCP).

Also, specifically for 7Zip: user ray023 answers this question in the SuperUser Stack Q&A: https://superuser.com/a/422463

Basically, if instead of drag-dropping your file you use the "extract here" method available in bith 7-ZIP and Winrar, the files are directly extractly to the right directory.

Improve this answer
4
  • 2
    The question was not "Why does 7zip use temp files?" The actual question was "Why are temp files commonly used in software development?"
    – Phil
    Commented Jun 16, 2012 at 20:22
  • @Phil The example given is, from what I understand, what happens when you drag one file from an archive into a folder on the system, without explicitely asking to "extract it". Yet, I re-read the question, and yes, I think you are right in that the author wanted to know more about the utility of temporary folders, which by the way was masterfuly answered by tdammers (+1 voted by me too)
    – Jalayn
    Commented Jun 16, 2012 at 23:11
  • While your answer wasn't exactly what i was looking for, +1, for solving my problem with 7zip
    – Devdatta Tengshe
    Commented Jun 17, 2012 at 11:42
  • 1
    Good comment basically boils down to: why does a program use a temp folder? Because when drag and dropping it doesn't know the destination folder and just "gives it" to windows, then windows takes over and puts the file in the right place.
    – Pieter B
    Commented Jun 29, 2012 at 8:06
0

If you have to do any kind of data processing to the file (decode/convert/etc..), then it's better to use a temporary file and when completed, and only if completed, transfer the result to the final destination.

Benefits:

  1. Only completed files reach destination
  2. Temporary file may (should) reside in fast media
  3. Avoid fragmentation on final file
  4. Allows the use of other media as final destination (ftp, cloud, whatever)
  5. Aborted temp files are easier to clean

I don't see real benefits of writing straight to the destination while processing the data.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.