66

FAT16 uses 16 bits to identify clusters. Thus there are a maximum of 65536 clusters before you run out of identifiers, and some identifiers are reserved for non-file uses. Each file occupies at least one cluster. Larger clusters increases the minimum allocation per file, increasing the overhead of small files. The cluster size then tells you the maximum ...


13

Use a database, thats what they are for. File storage has its place, but I wouldnt use it for this kind of scenario. Consider, for example, getting a list of blog posts containing a certain tag. Doing that with a database is trivial - likely just a single SQL statement. Doing it with files will involve a lot of file manipulation.


12

There were actually several variants of "FAT16" over the years with different limits but lets consider the version that persisted from "compaq DOS 3.31" through to windows 95. FAT volumes are divided into clusters. Each cluster is made up of a power of two number of sectors. The number of sectors per cluster on FAT16 is stored as an 8 bit signed number. So ...


9

Read speed and caching is an important factor, but it's not the only factor, and perhaps not even the primary factor in selecting a filesystem's block size. Every block on your filesystem has overhead associated with it. The filesystem must track which blocks are free, which blocks belong to which files, etc. This overhead must be stored on the disk ...


7

The reason there are many different file formats is that there are many different goals for the way data is formatted. Some of these are in opposition to each other and some are orthogonal to each other. Before you can embark on this, you need to determine what goals you wish to achieve. I would say the first and most prominent decision is human-...


7

NO, don't store files in a relational database Trust me, I've learned this the hard way. One problem with applications that deal with files, is as they evolve, the users always want to store more than the application was intended to handle. I once created an application with a document storage component meant to store Word and Excel documents. The storage ...


7

Most file system limits how many files you can have in a directory, which means you are forced to use subdirs anyway if the amount can grow indefinitely. Beware there is a limit on number of dirs as well (which is usually lower, say 32000 on ext3) If you can calculate the exact path to a file then there is no performance problem reading that file. Listing ...


7

Because we want our computers to do more than just execute one fixed program. Having a stream of opcodes in memory and endlessly stepping through it is fine if you have a single-purpose machine. But on a real computer, you want to achieve things such as have one binary call another and transfer control to it have one binary call another and get control ...


7

This is actually a very modest number of files for a doc management system. 5200 files x 52 weeks x 10 years is less than 3 million. Even at your own calculation, its only 1.5 TB of data over 10 years. That will easily fit on a hard drive. For this volume of files, I would recommend keeping the files in the file system, not the database. It will give you ...


6

Because reading multiple blocks allows the OS to cache the next few blocks and avoid the latency when the application eventually needs them. The biggest slowdown in accessing spinning disk drives (HDD and optical disk) is the seek time, literally the reading head moving to where the track with the block is. This is alleviated with solid state but reading ...


5

Some experimental operating systems don't even have any files. They have some orthogonal persistence machinery. Look at some academic OS projects (Coyotos, Grasshopper, IsaacOS, etc...). And old Lisp Machines in the 1980s might also had no file systems as we know them today. The inactive tunes.org site had some discussions (from the previous century) about ...


5

I think the folder based, tree-like file system is common but it's not the best. Indeed i believe a good classification of a file is better than to place a file in a certain 'place' e. g. folder. Files are different in content and so, a mp3-file contains different meta information compared to let's say a png-file. Shown in a list with columns a problem ...


5

Direct answer Is it safe to store application files to user's temp directory? To the letter of your question, yes, it is safe. However, very much take the bolded part into account. Not just any temp directory, only the user's temp directory suffices because it is appropriately access controlled to only give access to the current user. os.tmpdir() gives ...


3

Obviously, you have some options but there are also trade-offs. To answer your questions: Is this a reasonable way to store files of this size for serving? The answer of that really depends on your database and the recommendations for that. For example SQL Server has good support for arbitrary sized files if you use the FILESTREAM option (or something ...


3

You could compose the document ID like yywwdcxxxx yy last two digits of the year ww the calendar week 1..53 d the day 1..7 c check sum digit from ID+filename (cf. http://en.wikipedia.org/wiki/Check_digit) xxxx sequence number to distinguish documents from the same day Directly map this ID structure to a directory structure in a filesystem. ...


3

The idea of hashing the directories (not to be confused with generating a hash of the file name) and thus breaking it down into multiple directories is a good one. If its multi-tenant, you probably have a database id for the clients. And its a good idea to track the existence of the files in your db with a unique id (auto-increment or other). So you might ...


3

The problem you mention doesn't exist, since the limit you specified in your question is wrong. Until Vista, Explorer was limited to 248 characters. .NET Framework File is limited to 260 characters (MAX_PATH). The real maximum path for Windows itself is approx. 32,767 characters. I have no idea where have you found the mention of 255 characters. Now, if ...


2

You don't seem to picture the design of your system very clearly yet. So a bit of advice: Make it as simple as possible. Write down several scenarios of interaction between the user and your program. Be careful to include cases when the user changes his mind in the middle of saying a command, and other scenarios when things go wrong. Don't proceed until you ...


2

That sounds like you would be reimplementing rsync in Python. If you're mainly synchronizing NIX machines you should talk a look at using that. Specifically for Python, libraries like fabric can help you push files to remote machines.


2

See the MSDN article on maximum path length limitations for some additional ideas. You can try some of the following depending on what is available to you: Attempt to use "extended-length paths" starting with \\?\ but be aware of the limitations such as no relative path names. Map a longer path to a drive using subst to artificially shorten the path. If ...


2

Whenever you read from or write to the buffer, only it's position changes. So you are wrong about reading from channel. Note that reading from channel means writing to the buffer. You can set the position to 0 and limit - to the maximum number of bytes to read before you call Channel.read(ByteBuffer). The read operation then updates the position to the ...


2

Either put the document in a central location (database or shared file system), using unique file names or identifiers for each document created, or recreate the document on each server, whenever it is found to be missing.


2

Clearly, you have to make server requests at some point to get the user's stuff. For optimal UX, assuming it's impractical to load everything upfront, it'd be silly not to use some kind of AJAX request. So the real question is: What should the granularity of your requests be? The correct answer has more to do with UX than code quality, and will depend very ...


2

This is exactly what premature optimization is about. Do the math. 31 MB for two days means 5.5 GB per year, which is absolutely nothing. Don't you have more important stuff to worry about, like, I don't know, the indexes? This doesn't mean you shouldn't try to minimize disk usage when it's easy to do. You had an excellent idea to enable compression: in one ...


2

While simply uploading the files to a web server and allowing access to them is an option. There are a couple of potential requirements which you should consider. 1: authentication of the user requesting the file. 2: accessing files by fields other than thier filename. This kind of thing can steer you towards the approach you seem to be taking. Ie 1: ...


2

Key value stores that don't persist to disk are generally billed as caches, implying there is another long-term durable storage your application manages somewhere else. If your data is important to keep long-term, a cache won't help you, even if distributed. You want to write it to disk, and you want to protect against corruption and application mistakes by ...


2

There is no point. Wow, talk about a "zen" answer. However, what I mean is that instead of a point, in reality you rather have a spectrum. Hard criteria Don't get me wrong, a hard criterium like your "the dataset cannot fit into memory", you were correct in it being "necessary" at that point. The same goes for other "hard&...


2

It Depends... If you are really interested in all of the possible ways find yourself a book on database engines, something like An Introduction to Database Systems The Dumb Way Treat every record as an exact length record, with each column being located at some bit-offset and consuming some X bits of the record. You could pack this record tight, or align ...


1

I figured it out, i was over complicating the whole thing, i will just append the metadata on the filename, this way when i call StorageFolder.GetfilesAsync() i will get all the folder-paths and then extract the metadata from them, best part is that This getfoldersasync Function does not open up a stream against them. After that i can always save them to ...


Only top voted, non community-wiki answers of a minimum length are eligible