EDIT: You were right, the problem is not checking if a file with the same name exists but the connection. I made a mistake measuring the time with StopWatch and included a part which used the network connection.

Original Question: I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have children which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

  • 2
    You have neglected to mention the most important thing: give us a rough idea about the number of files that you will need to have in your cache. – Mike Nakis Sep 20 '15 at 13:58
  • 4
    Also, please post your code which checks if the file exists in the cache. I find it hard to believe that you have a problem with the time it takes to conduct a simple search to see if a file with a certain filename exists in a folder. – Mike Nakis Sep 20 '15 at 14:00
  • 2
    +1 - unless you have a massive number of files in the same directory (probably tens of thousands), checking for existence should be a trivial operation. If you have more files than this, structuring them in predictable subdirectories may be necessary. – Jules Sep 20 '15 at 15:57
  • 2
    Also of interest: did you measure the individual parts of your program to be sure it needs the most time where you believe it does? I am here with Mike Nakis, I find it hard to believe that the bottleneck is where you described it. – Doc Brown Sep 20 '15 at 16:18
  • 1
    @JoeFox: well, this is up to you, but deletion would remove the question from "Programmers" front page, saving others the time to read your question again just to note at the end that this has become obsolete. However, I was under the impression your original question might have been improved to become a good fit for this site. – Doc Brown Sep 21 '15 at 9:32

Saving stuff to files isn't normally a good way of creating a cache. If in your case its the downloading of large files over the network from a remote machine which is slow. I would create another identical local database to store the files once you have downloaded them.

If this is not possible, create a small database in which you just store the id, filename and directory of the downloaded files. You can use this to check for the existence of the file and then read it off disk as you do now.

Rather than reading all the lines in a file to see if it's the same as the new one, create a checksum when you download it and add this to the database table.

| improve this answer | |
  • Does not make much sense to me. Putting files into a local database instead of a file system will slow down the process even more. A database will only make sense if one needs to associate additional metadata to the files not provided by the file system itself. And from the OP's Dictionary approach one can assume there is no need for calculating checksums (which requires to scan each file at least once), the unique identifiers seem to be the file names (but I am still expecting a clarification from the OP). – Doc Brown Sep 21 '15 at 6:16
  • An evolving question. File.Exists() is pretty slow in my experience, then theres the 300k files which would be too mamy for a single directory. Then the scanning of files, which I assumed was to check for differences when modifing the origional database was not possible – Ewan Sep 21 '15 at 8:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.