In-memory data collection object(s) vs database

Question

I have a couple of questions about the practicality of developing a WPF application that loads data into memory from a text file and then manipulates the resulting object(s) instead of transactions with a database such as SQLite. Multiple views would have bindings on the resulting object(s) in memory. Persisting the updates to the text file on disk might only happen at app shutdown.

The data is a ham radio logbook stored as .adi format. ADI is a textual format that stores tabular data in strings, including column names and their values. Rows of data are terminated with an tag. There are numerous apps in existence that parse these .adi files line by line into a database table where it is read and edited by the app. The entire database can then be exported back to an .adi file whereby each table record is serialized into a line of text in the .adi file. Most of them are utilizing some form of database like SQL.

The largest logbook that I'm aware of is about 120,000 records, but a typical user might reach 60,000 to 70,000 in their lifetime. Each record of data could have as many as 30-40 columns including a mixture of string, integer and datetime values.

I'm interested in developing an app that can handle say 100,000 such records without the use of a database solution such as SQLite or MS SQL. An .adi text file of this magnitude should be under 50mb in size on disk. When the app first loads, it would read the text file into an object for binding to a grid. Edits would be applied to the object(s) until app shutdown when they are persisted to the file on disk.

1. Is an object based on a text file approximately 50mb in size likely to be problematic if the app is running on a semi-modern Windows machine with 4gb-8gb of system memory?

2. The ADI standard includes many enumerations. If I were converting this file standard to a relational database I would have numerous lookup tables representing the enumerations. But given the goal of developing without the need for an external database solution, I am thinking of storing the enumerations as class objects in a library. Am I correct that a hard-coded dictionary or enumeration does not occupy any memory space until it is called?

Thanks for your help.

Answer 1

Is an object based on a text file approximately 50mb in size likely to be problematic if the app is running on a semi-modern Windows machine with 4gb-8gb of system memory?

Not the object itself. I'm more concerned with the way you plan on implementing the UI. The grid design you propose will require many, many times the amount of memory that holding the actual ADIF will require, and it won't exactly be user-friendly.

Am I correct that a hard-coded dictionary or enumeration does not occupy any memory space until it is called?

You can certainly make such objects lazy-loading. However, I've had a look at the enumerations, and the amount of memory required for them is almost certainly inconsequential anyway.

...without the use of a database solution such as SQLite or MS SQL

There are many compelling reasons why those other guys built their apps on top of a database system; you get loads of capabilities for free. Having all that log data in a database suddenly makes it useful for all sorts of purposes, not the least of which is filtering, searching, sorting and querying the resulting data.

Answer 2

I agree with the other answers here but I want to add a little detail as to why building this all in-memory and persisting to a file can be problematic.

If you were just building an application to view these files, then building a DB for it would probably be overkill. But you want to make edits and then save everything at the end. This approach is problematic:

• If the application crashes (computer shuts down, user forces it down, etc.), all changes since the last time it was opened will be permanently lost.
• If the application crashes during the save or has an bug in the save routine, you will corrupt the file and potentially lose the entire thing.

I've had the displeasure of working with applications that save changes on exit and I hate that about them. I consider such an approach a bad practice. To fix it with a file design, you need to, at the very least, back up the file before you attempt to write it. Then you'll need to be able to save the file either on change or let the user control when the save is done. Saving the entire file could be a little slow but the files are not very big so it might be OK. That's your minimal implementation.

If you want to make this better, though, you'd need to be able to update sections of the file without writing the entire thing every time. Once you get to this point of sophistication, using a database is much easier than trying to manipulate a file in this way.

Answer 3

If you are talking .NET a single object or collection is limited to 2 GB and you don't get all of that as it needs to be contiguous. If it is only 50 MB on disk then you should be OK.

As Robert said - if your design is to display it ALL in one grid that is going to be more memory than the data. You may want to rethink that design.

You can certainly use collection and enumerations to store your data in memory.

You can search collections with LINQ.

I wrote small application where data was read from XML, manipulated in memory, and then written back to XML. But it would have been a lot easier to write it as a database application. Data in XML was a requirement.