140

Reasons in favor of storing files in the database: ACID consistency including a rollback of an update which is complicated when the files are stored outside the database. This isn't to be glossed over lightly. Having the files and database in sync and able to participate in transactions can be very useful. Files go with the database and cannot be orphaned ...


140

Temporary files have to be stored into the operating system temporary directory for several reasons: The operating system makes it very easy to create those files while ensuring that their names would be unique. Most backup software knows what are the directories containing temporary files, and skips them. If you use the current directory, it could have an ...


96

A few reasons I can think of: On most platforms, file moves are atomic, but file writes are not (especially if you can't write all the data in one go). So if you have the typical producer/consumer pattern (one process produces files, the other watches a directory and picks up everything it finds), writing to a temp folder first and only then moving to the ...


91

In many cases, this is a bad idea. It will bloat the database files and cause several performance issues. If you stick the blobs in a table with a large number of columns it's even worse. However! Some databases, like SQL Server have a FILESTREAM column type. In this case, your data is actually stored in a separate file on the database server and only ...


40

Calculating CRCs (or, better, sha1sums) on both files requires reading every byte anyway. If you do a byte-by-byte comparison, you can quit as soon as you see a mismatch -- and you don't have to worry about two different files that happen to have the same checksum (though that's vanishingly unlikely for sha1sum). So if you're doing the comparison locally, a ...


39

Convenient interfaces are nice, and sometimes the way to go. However, most of the time good composability is more important than convenience, as a composable abstraction allows us to to implement other functionality (incl. convenience wrappers) on top of it. The most general way for your function to use files is to take an open file handle as parameter, as ...


35

Yes, it is a bad practice. Performance impact on the DB: if you do a SELECT with any BLOB column, you will always do a disk access, while without BLOBs you have a chance to get data straight from RAM (high throughput DB will be optimized to fit tables in RAM); replication will be slow, replication delay high, as it will have to push BLOB to slaves. High ...


33

Should I insist saving to /tmp is the right approach and defend for any failure as "working as intended" (ie. ask your admin for proper permission access)? There are standards for this, and the best thing you can do is conform to them. POSIX, which is followed by pretty much every non-mainframe OS of any significance that you're likely to run into, has ...


27

You should try to meet two goals: Uniqueness, and usefulness. Using a GUID guarantees uniqueness, but one day the files may become detached from their original source, and then you will be in trouble. My typical solution is to embed crucial information into the filename, such as the userID (if it belongs to a user) or the date and time uploaded (if this is ...


22

The real question is one of completeness. Is your file processing function the complete processing of the file, or is it just one piece in a chain of processing steps? If it is complete in and of its own, then feel free to encapsulate all file access within a function. def ver(filepath): with open(filepath, "r") as f: # do processing steps on f ...


20

I would be pragmatic about it, and follow the "don't optimize yet" principle. Make the solution that makes sense at the moment, and one that you have the development resources to properly implement. There are plenty of potential problems. But those do not necessarily become real problems. E.g. It would probably not be a problem if you have 100 users. It ...


19

This depends on what the limiting factor is, doesnt it? If the bottleneck is the utility program, then sure, running more than one copy or using more threads will speed things up. If the network is the limiting factor, then adding multiple instances of the utility isnt going to help since you still will be stuck moving at most X bytes per second. In fact ...


19

File pointers are not the only way to access files on POSIX systems (which, obviously Linux is). As a matter of fact, the FILE struct exists to make it easier to work with files in a cross-platform manner (fopen() will work on both Linux, Mac OS X, any BSD and Windows) and to provide some facilities like transparent buffering (buffered I/O is way faster). ...


19

First, lets talk about what your goals are: you obviously don't want to test "file formats" - you want to to test your different FileReader implementations you want to find as many different types of errors as possible by automatic tests To reach that goal in full, IMHO you have to combine different strategies: first, real unit testing: your FileReader ...


16

Microsoft released a white paper about this a few years back. It concentrates on SqlServer, but you may find some interesting information in there: To BLOB or not to BLOB? Large Object Storage in a Database or a Filesystem? A very concise version of their conclusion is: When comparing the NTFS file system and SQL Server 2005, BLOBS smaller than 256KB ...


16

Using fscanf() in itself probably explains most of it. fscanf() has to interpret the passed-in format string, and then has to scan the input stream from the file, trying to match the specified pattern. That's actually a huge amount of work. read() just has to read in the specified number of bytes from the file and doesn't have to do any parsing of the input. ...


16

Switching to a solution involving databases or the queuing systems mentioned by Ewan would create dependency on a new, complex system in both backend and frontend introduce unnecessary complexity and a sh*tload of new points of failure increase cost (including cost of ownership) Moving/renaming files within a single volume is guaranteed to be atomic on all ...


15

This seems to be an issue in Windows, more specifically related to how the drag-drop is managed. The developers of the WINSCP client have developed their own shell extension, which overrides this drag-drop behavior and allows dropping the file to the right folder immediately. They explain the trick in their documentation and, more interesting, what is the ...


15

No, there isn't. Slashes work everywhere, backslashes work only on Windows and are a pain to type and read. They are used only by people who mistakenly think they have to use them. But of course, you should really be using the Paths API (documentation for JDK8, JDK9) and never write explicit path separators anyway.


15

When seeing a class with a constructor signature like EnglishWordsListGenerator(const std::string &wordFileName) I think it is pretty obvious that this constructor will read the given file (and so need some time), and it should not be to hard to understand that the caller has to care for possible exceptions from this (because file IO can fail). So ...


15

multipart/form-data is a construct created for HTML forms. As you've discovered the positive of multipart/form-data is the transfer size is closer to the size of the object being transferred--where in a text encoding of the object the size is inflated substantially. You can understand that internet bandwidth was a more valuable commodity than CPU cycles when ...


14

The most likely efficient way to do this is: Have a single thread that reads the input file. Harddisks are at their fastest when reading sequentially. Do not read it into memory all at once! That is a huge waste of memory which could be used much better to speed the processing! Instead, have this single thread read a bundle of entries (maybe 100, maybe 1000,...


13

As you do always in TDD with external resources: you create one or more interfaces to your filesystem operations and "mock them out". You want to test your "table generator" and your meta-data modification code, not the file system operations itself (most probably you are using ready-made library implementations for accessing the file system).


12

If you are working in a language that supports it, I'd provide a Save method that takes a Stream. That way, the user can save the data whereever he or she wants. It takes 20 seconds longer to write than saving just to a file, but it is easily understood by a programmer, and at the calling site it is very clear what actually happens. The way you described ...


11

How multithreading will not help: Multiple threads simultaneously reading from the client's disk or simultaneously sending stuff over the network will not help at all, since there most likely exists only one path of communication between the client and the server, the client is most likely reading files off from a single hard-drive, and the files are most ...


11

The old conventional wisdom of storing files outside database might no longer hold. As a matter of principle, I'd favor integrity over speed, and with a modern DBMS, you can have both. Tom Kyte seems to agree: I know of no advantages to storing data I want to keep for a long time outside of a database. If it is in the database I can be sure it ...


11

Whats wrong with having a "test" file system? Create a template folder/directory structure which has enough content to test you operations. During setup of your unit test copy this initial structure (would recommend you ZIP up the template and unzip into your test area). Run your tests. Delete the whole thing during tear down. The problem with mocking is ...


10

Here's another way to think about it. If there is no possibility that two different files have the same CRC, then by extension it means that every file can be represented by a unique CRC.If the CRC was smaller than the original file then it would represent a form of lossless compression. If not, you would do just as well to compare the original files since ...


10

Both options are over-engineered, involving the database is inappropriate here, and the directory structure is too deep. Instead, for administrator-friendly software, I see the following patterns emerging: Paths are configured in config files, not database. Usually, you want to be able to copy the database dump to another system. Often, the database ...


10

You might have a look at the PNG file format and how it handles version compatibility. Every block has an id describing what kind of block it is, and it has some flags that tell the software what to do if it cannot understand that id. For example "you cannot read the file if you don't understand this block", or "you can read the file but not modify it", or "...


Only top voted, non community-wiki answers of a minimum length are eligible