Post Closed as "unclear what you're asking" by Doc Brown, user22815, durron597, Ixrec, gnat
3 added 244 characters in body
source | link

EDIT: You were right, the problem is not checking if a file with the same name exists but the connection. I made a mistake measuring the time with StopWatch and included a part which used the network connection.

Original Question: I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have children which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have children which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

EDIT: You were right, the problem is not checking if a file with the same name exists but the connection. I made a mistake measuring the time with StopWatch and included a part which used the network connection.

Original Question: I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have children which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

2 Remove signature. Detag title. Remove resource request.
source | link

C# Caching Improving performance with caching of files - Performancein C#

I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • Is there an open source file cache implementation with high performance?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have childschildren which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

Thanks in advance and kind regards, Fox

C# Caching files - Performance

I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • Is there an open source file cache implementation with high performance?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have childs which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

Thanks in advance and kind regards, Fox

Improving performance with caching of files in C#

I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have children which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

1
source | link

C# Caching files - Performance

I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.

My ideas until now:

  • Lazy loading - just download the file if it could not be accessed
  • Group the files in folder so that I have less files in one directory
  • Creating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)

I invested already some time and before I reimplement it I want to know:

  • Is there something else (idea) which I could try?
  • Is there an open source file cache implementation with high performance?
  • In C# is there a best practice regarding file handling which leads to the best performance?

Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?

We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have childs which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call "file cache".

Thanks in advance and kind regards, Fox