96

A few reasons I can think of: On most platforms, file moves are atomic, but file writes are not (especially if you can't write all the data in one go). So if you have the typical producer/consumer pattern (one process produces files, the other watches a directory and picks up everything it finds), writing to a temp folder first and only then moving to the ...


66

FAT16 uses 16 bits to identify clusters. Thus there are a maximum of 65536 clusters before you run out of identifiers, and some identifiers are reserved for non-file uses. Each file occupies at least one cluster. Larger clusters increases the minimum allocation per file, increasing the overhead of small files. The cluster size then tells you the maximum ...


15

This seems to be an issue in Windows, more specifically related to how the drag-drop is managed. The developers of the WINSCP client have developed their own shell extension, which overrides this drag-drop behavior and allows dropping the file to the right folder immediately. They explain the trick in their documentation and, more interesting, what is the ...


13

Use a database, thats what they are for. File storage has its place, but I wouldnt use it for this kind of scenario. Consider, for example, getting a list of blog posts containing a certain tag. Doing that with a database is trivial - likely just a single SQL statement. Doing it with files will involve a lot of file manipulation.


12

There were actually several variants of "FAT16" over the years with different limits but lets consider the version that persisted from "compaq DOS 3.31" through to windows 95. FAT volumes are divided into clusters. Each cluster is made up of a power of two number of sectors. The number of sectors per cluster on FAT16 is stored as an 8 bit signed number. So ...


9

Read speed and caching is an important factor, but it's not the only factor, and perhaps not even the primary factor in selecting a filesystem's block size. Every block on your filesystem has overhead associated with it. The filesystem must track which blocks are free, which blocks belong to which files, etc. This overhead must be stored on the disk ...


7

Because we want our computers to do more than just execute one fixed program. Having a stream of opcodes in memory and endlessly stepping through it is fine if you have a single-purpose machine. But on a real computer, you want to achieve things such as have one binary call another and transfer control to it have one binary call another and get control ...


7

This is actually a very modest number of files for a doc management system. 5200 files x 52 weeks x 10 years is less than 3 million. Even at your own calculation, its only 1.5 TB of data over 10 years. That will easily fit on a hard drive. For this volume of files, I would recommend keeping the files in the file system, not the database. It will give you ...


7

Most file system limits how many files you can have in a directory, which means you are forced to use subdirs anyway if the amount can grow indefinitely. Beware there is a limit on number of dirs as well (which is usually lower, say 32000 on ext3) If you can calculate the exact path to a file then there is no performance problem reading that file. Listing ...


7

The reason there are many different file formats is that there are many different goals for the way data is formatted. Some of these are in opposition to each other and some are orthogonal to each other. Before you can embark on this, you need to determine what goals you wish to achieve. I would say the first and most prominent decision is human-...


7

NO, don't store files in a relational database Trust me, I've learned this the hard way. One problem with applications that deal with files, is as they evolve, the users always want to store more than the application was intended to handle. I once created an application with a document storage component meant to store Word and Excel documents. The storage ...


6

Because reading multiple blocks allows the OS to cache the next few blocks and avoid the latency when the application eventually needs them. The biggest slowdown in accessing spinning disk drives (HDD and optical disk) is the seek time, literally the reading head moving to where the track with the block is. This is alleviated with solid state but reading ...


5

Some experimental operating systems don't even have any files. They have some orthogonal persistence machinery. Look at some academic OS projects (Coyotos, Grasshopper, IsaacOS, etc...). And old Lisp Machines in the 1980s might also had no file systems as we know them today. The inactive tunes.org site had some discussions (from the previous century) about ...


5

I think the folder based, tree-like file system is common but it's not the best. Indeed i believe a good classification of a file is better than to place a file in a certain 'place' e. g. folder. Files are different in content and so, a mp3-file contains different meta information compared to let's say a png-file. Shown in a list with columns a problem ...


5

Direct answer Is it safe to store application files to user's temp directory? To the letter of your question, yes, it is safe. However, very much take the bolded part into account. Not just any temp directory, only the user's temp directory suffices because it is appropriately access controlled to only give access to the current user. os.tmpdir() gives ...


3

You could compose the document ID like yywwdcxxxx yy last two digits of the year ww the calendar week 1..53 d the day 1..7 c check sum digit from ID+filename (cf. http://en.wikipedia.org/wiki/Check_digit) xxxx sequence number to distinguish documents from the same day Directly map this ID structure to a directory structure in a filesystem. ...


3

The idea of hashing the directories (not to be confused with generating a hash of the file name) and thus breaking it down into multiple directories is a good one. If its multi-tenant, you probably have a database id for the clients. And its a good idea to track the existence of the files in your db with a unique id (auto-increment or other). So you might ...


3

The problem you mention doesn't exist, since the limit you specified in your question is wrong. Until Vista, Explorer was limited to 248 characters. .NET Framework File is limited to 260 characters (MAX_PATH). The real maximum path for Windows itself is approx. 32,767 characters. I have no idea where have you found the mention of 255 characters. Now, if ...


2

That sounds like you would be reimplementing rsync in Python. If you're mainly synchronizing NIX machines you should talk a look at using that. Specifically for Python, libraries like fabric can help you push files to remote machines.


2

See the MSDN article on maximum path length limitations for some additional ideas. You can try some of the following depending on what is available to you: Attempt to use "extended-length paths" starting with \\?\ but be aware of the limitations such as no relative path names. Map a longer path to a drive using subst to artificially shorten the path. If ...


2

Whenever you read from or write to the buffer, only it's position changes. So you are wrong about reading from channel. Note that reading from channel means writing to the buffer. You can set the position to 0 and limit - to the maximum number of bytes to read before you call Channel.read(ByteBuffer). The read operation then updates the position to the ...


2

Storing large binary blocks as files is typically more efficient then storing large BLOBs in a database. It depends. GUIDs have the advantage you can create one at random and use it without depending on some identity provider. Using a seed based ID generated in a DBMS would require you to first go to the database before writing a file to disk, with a GUID ...


2

I don't see any diff'ing or comparing between the files, so there's no need to copy them around or even load them into RAM. My suggestion is: deal with the files individually. And if you want it to be really flexible, consider storing then in a database. If that's not feasible, at least only write the changed file to your storage. Store where's the newest ...


2

You don't seem to picture the design of your system very clearly yet. So a bit of advice: Make it as simple as possible. Write down several scenarios of interaction between the user and your program. Be careful to include cases when the user changes his mind in the middle of saying a command, and other scenarios when things go wrong. Don't proceed until you ...


2

This is exactly what premature optimization is about. Do the math. 31 MB for two days means 5.5 GB per year, which is absolutely nothing. Don't you have more important stuff to worry about, like, I don't know, the indexes? This doesn't mean you shouldn't try to minimize disk usage when it's easy to do. You had an excellent idea to enable compression: in one ...


2

While simply uploading the files to a web server and allowing access to them is an option. There are a couple of potential requirements which you should consider. 1: authentication of the user requesting the file. 2: accessing files by fields other than thier filename. This kind of thing can steer you towards the approach you seem to be taking. Ie 1: ...


2

Either put the document in a central location (database or shared file system), using unique file names or identifiers for each document created, or recreate the document on each server, whenever it is found to be missing.


2

Clearly, you have to make server requests at some point to get the user's stuff. For optimal UX, assuming it's impractical to load everything upfront, it'd be silly not to use some kind of AJAX request. So the real question is: What should the granularity of your requests be? The correct answer has more to do with UX than code quality, and will depend very ...


Only top voted, non community-wiki answers of a minimum length are eligible