4

My C# .NET application has to read lots of files with electrical readings. There are several calculation to be done and output files need to be generated. Due to the nature of the input it would be very inefficient to output after each input line or even after one file, so it would be better to do so after a chunk of files.

The reason for this is, that each file contains a list of readings/electrical units for one time stamp, but the output needs to be files for each unit.
So for instance the input would be 100 files each containing e.g. volt, ampere, watt, status code for one timestamp (2015_08_31_00_00_00.txt, 2015_08_31_00_00_05.txt ...).
The output should be per electrical unit and e.g. one file per day
(c://ampere/2015_08_31.txt, c://volt/2015_08_31.txt ...).

Note that this is a simplified picture of the application. In reality there are several different input and output formats, directory structures etc.

What I want to do is, keep the calculated and ready for output values in memory and output them according to different strategies. This could be e.g. until all input files are processed.

As some strategies like this one can result in too much data in memory, I would like to monitor the memory usage and decide if I need to output sooner.

I do know how to get the used memory of my program, but how do I safely get an estimation on the available memory?

As far as I understand this is not a simple task due to memory fragmentation, paging, trashing and so on. I do know in this case that I got the 32-Bit induced memory limit with the current build settings, but I would like to get a general answer which could also be applied to a 64-Bit program.

How do I estimate when enough memory is still available to write to disk, but use a good amount of RAM to optimize I/O?

Update
I have not yet implemented this feature so I cannot tell if memory problems would occur under usual circumstances. Thanks to all your comments and answers I see it isn't easily accomplishable.

My current idea is to use a fixed input limit based on the size of the input files. This still would not protect against problems if other programs creates heavy load. Each file is always just a few KB.

I will keep the SQLite idea in mind but I will have to check if I get an OK

Improve this question
4
  • See also: stackoverflow.com/q/2342023
    – rwong
    Sep 1, 2015 at 1:01
  • Yes I am aware of this post and know how to get the current memory usage (I did my research). But this does not tell my much about when I will face issues with memory running low as there are other applications running as well.
    – John
    Sep 1, 2015 at 1:03
  • @John Although if I understand your question around available memory correctly, you might be interested in checking the FreePhysicalMemory on the Win32_OperatingSystem WMI object (Example 1 and 2). Although it might be easier just to provide a configuration setting for your program to set the limit at which it stops growing, rather than managing it dynamically in code. That said, I'd still seriously consider the other suggestions, I think they provide simpler alternatives.
    – Daniel B
    Sep 1, 2015 at 6:14
  • One little help may be WeakReferences; object referenced by those can be freed by the GC. Of course, you should use those only in objects you can rebuild when needed (trading memory for the cost of building the object new when you need it).
    – SJuan76
    Sep 1, 2015 at 21:58

3 Answers 3

Reset to default
8

Unfortunately, there's really no good way to answer the question "how much RAM is my program using?" or "how close am I to hitting the OutOfMemoryException wall?", for a number of reasons. It's a lot more complicated than it looks like it should be.

One thing you might want to do, though, is make sure that you're not holding on to the input data unnecessarily. For example, depending on how you're processing the data, it could be feasible to read the file, and perform the relevant calculations, one line at a time, or one small batch of lines at a time. If you hold on to aggregate results, but don't keep the input data around, that should keep the memory usage down.

If that doesn't help, because your results just keep piling up, the best way to handle that is to offload them out of memory. Put them in a relational database, for example, as fast as they come in, and then they end up on disc instead. And then at the end of the day, perform a query that will pull in the day's results (HINT: this is a lot easier if you have a timestamp field that gets set when you insert the records) and save them out to their final version in your output files.

Improve this answer
12
  • I do not keep the input data in memory after the calculations are done. This feature with the new strategies is not yet implemented, but I don't think I would necessarly run out of memory. I just don't like to have no check on whether it might happen. The database is unfortunatley not an option I already had that idea when the requirements were made and it was declined.
    – John
    Sep 1, 2015 at 1:35
  • 3
    Why do you think "The database is unfortunatley not an option" ? Using a lightweight DB like SQLite is not much different from using a file store. The use case you are describing sounds pretty much like a typical application for an aggregate database. A database will also enforce you to think about a good data model for your problem, which is actually where I would start - think about where the memory consumption really happens before asking for different solutions.
    – Doc Brown
    Sep 1, 2015 at 5:53
  • 1
    @John it's impossible to answer the question because, due to the way operating systems work, there is no such figure. The operating system can report several different figures for memory usage (working set, virtual size, private allocation size) but none of these is "total amount of RAM used by the process" because RAM is used by the operating system to satisfy the requirements of all running processes in a fashion that means memory usage is not necessarily attributable to a single process.
    – Jules
    Sep 1, 2015 at 7:45
  • 1
    Memory can be shared between processes, which is hard to account for, and worse still memory can be used for system level caches that don't get attributed to any process, but those caches are still required in order to get reasonable performance, and the quantity of cache required depends in unpredictable ways on the applications running. Therefore, the techniques described in rwong's link provide figures that are, at best, estimates, and which could in fact be completely misleading.
    – Jules
    Sep 1, 2015 at 7:46
  • 6
    @John: SQLite does not require any installation, it does not require to buy a license, and if you use Firefox as your web browser, you already have a program at your PC which uses it. If your supervisor works with you at the code level, discuss the memory problem with him and what better idea he has than using SQLite. If he does not work with you at the code level, don't tell him the technical details, he won't even notice. Just tell him "we write the data into an intermediate, binary file".
    – Doc Brown
    Sep 1, 2015 at 16:05
2

you can use a performance counter from the System.Diagnostics namespace.
Use the two-string constructor with a categoryName of "Memory" and a counterName of "Available Bytes". You can then use something like
long availableBytes = Convert.ToInt64(myPerformanceCounter.NextValue()) to get a reading.
If you need more info than that provides then you can play around with different categories and counters.
Important Note: the PerformanceCounter class utilises the IDisposable interface so be sure to either wrap it in a using statement or call the Dispose() method when you are done.

Improve this answer
0

If you know the maximum amount of space you will need for your output, and it fits into your address space, you can use a function such as VirtualAlloc() to reserve a range of memory addresses, with or without committing pages. This won’t allocate any physical pages until you actually try to write to them. Therefore, you won’t get any out-of-memory errors just for reserving the memory region. You can also open the input files as memory-mapped files, so that the OS will only copy the physical pages from disk when you read them.

At that point, you won’t need to make any additional calls to allocate your memory: when you run out of physical memory, the OS should just page out the pages you aren’t using to virtual memory, or you can VirtualFree() the pages after they’re written.

Improve this answer
3
  • I'm not convinced you gain much by doing it this way. The C# heap can be used to buffer file contents just as efficiently and much more easily than using native, unmanaged memory for the purpose, and memory mapping the files really gains little over reading them through a stream - it may be slightly more efficient if you can avoid copying the data into a buffer, but it seems unlikely that this difference would be noticeable in the case described in the question, and the added complexity of dealing with a memory mapped file is probably not worth the investment in time to write it.
    – Jules
    Sep 1, 2015 at 8:26
  • Granted the benefits of memory-mapping the input files is marginal, but this is how e.g. glib’s g_file_get_contents() works by default. If you’re reimplementing that yourself, it’s a little extra work to initialize, but after that, it’s equivalent to having the whole file read into a buffer, without needing to read the whole file into a buffer.
    – Davislor
    Sep 1, 2015 at 9:04
  • As for the main point, the way I read the OP is that he can’t just allocate everything from the heap, because he’s getting out-of-memory errors. My understanding—please correct me if I’m wrong—is that, if you allocate a large new C# object, it comes from the Large Object Heap, which internally uses VirtualAlloc(). If you don’t need to worry about how many pages are committed as private bytes, that is simpler. Pick a buffer size based on System.Management, output when the buffer is full, re-use? Let the OS decide when to page to disk?
    – Davislor
    Sep 1, 2015 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.