EDIT: You were right, the problem is not checking if a file with the same name exists but the connection. I made a mistake measuring the time with StopWatch and included a part which used the network connection.
Original Question: I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.
My ideas until now:
- Lazy loading - just download the file if it could not be accessed
- Group the files in folder so that I have less files in one directory
- Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)
I invested already some time and before I reimplement it I want to know:
- Is there something else (idea) which I could try?
- In C# is there a best practice regarding file handling which leads to the best performance?
Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?
We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have children which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".