My C# .NET application has to read lots of files with electrical readings. There are several calculation to be done and output files need to be generated.
Due to the nature of the input it would be very inefficient to output after each input line or even after one file, so it would be better to do so after a chunk of files.
The reason for this is, that each file contains a list of readings/electrical units for one time stamp, but the output needs to be files for each unit.
So for instance the input would be 100 files each containing e.g. volt, ampere, watt, status code for one timestamp (2015_08_31_00_00_00.txt, 2015_08_31_00_00_05.txt ...).
The output should be per electrical unit and e.g. one file per day
(c://ampere/2015_08_31.txt, c://volt/2015_08_31.txt ...).
Note that this is a simplified picture of the application. In reality there are several different input and output formats, directory structures etc.
What I want to do is, keep the calculated and ready for output values in memory and output them according to different strategies. This could be e.g. until all input files are processed.
As some strategies like this one can result in too much data in memory, I would like to monitor the memory usage and decide if I need to output sooner.
I do know how to get the used memory of my program, but how do I safely get an estimation on the available memory?
As far as I understand this is not a simple task due to memory fragmentation, paging, trashing and so on. I do know in this case that I got the 32-Bit induced memory limit with the current build settings, but I would like to get a general answer which could also be applied to a 64-Bit program.
How do I estimate when enough memory is still available to write to disk, but use a good amount of RAM to optimize I/O?
I have not yet implemented this feature so I cannot tell if memory problems would occur under usual circumstances. Thanks to all your comments and answers I see it isn't easily accomplishable.
My current idea is to use a fixed input limit based on the size of the input files. This still would not protect against problems if other programs creates heavy load. Each file is always just a few KB.
I will keep the SQLite idea in mind but I will have to check if I get an OK