2

I have a couple of questions about the practicality of developing a WPF application that loads data into memory from a text file and then manipulates the resulting object(s) instead of transactions with a database such as SQLite. Multiple views would have bindings on the resulting object(s) in memory. Persisting the updates to the text file on disk might only happen at app shutdown.

The data is a ham radio logbook stored as .adi format. ADI is a textual format that stores tabular data in strings, including column names and their values. Rows of data are terminated with an tag. There are numerous apps in existence that parse these .adi files line by line into a database table where it is read and edited by the app. The entire database can then be exported back to an .adi file whereby each table record is serialized into a line of text in the .adi file. Most of them are utilizing some form of database like SQL.

The largest logbook that I'm aware of is about 120,000 records, but a typical user might reach 60,000 to 70,000 in their lifetime. Each record of data could have as many as 30-40 columns including a mixture of string, integer and datetime values.

I'm interested in developing an app that can handle say 100,000 such records without the use of a database solution such as SQLite or MS SQL. An .adi text file of this magnitude should be under 50mb in size on disk. When the app first loads, it would read the text file into an object for binding to a grid. Edits would be applied to the object(s) until app shutdown when they are persisted to the file on disk.

  1. Is an object based on a text file approximately 50mb in size likely to be problematic if the app is running on a semi-modern Windows machine with 4gb-8gb of system memory?

  2. The ADI standard includes many enumerations. If I were converting this file standard to a relational database I would have numerous lookup tables representing the enumerations. But given the goal of developing without the need for an external database solution, I am thinking of storing the enumerations as class objects in a library. Am I correct that a hard-coded dictionary or enumeration does not occupy any memory space until it is called?

Thanks for your help.

6
  • It's actually the ADIF specification. Oct 4, 2016 at 2:08
  • 1
    The real question is: why would you want to do that? The database approach gives you so much for free - at the moment I only see tons of downsides for the proposed solution.
    – Eiko
    Oct 4, 2016 at 13:21
  • 2
    There is a C# version of SQLite with source code available that you could probably work into your own app without any further effort from installers, etc. That said, there are .NET clients that you can use with SQLite proper, and the deploy shouldn't be more than a small handful of files. You should never have to update SQLite; the core functionality has been stable for years. Oct 4, 2016 at 19:02
  • 1
    See also nuget.org/packages?q=system.data.sqlite Oct 4, 2016 at 19:08
  • 1
    I didn't realize implementing SQLite in a WPF application was as simple as adding the library and namespace. For some reason I assumed you had to download and install SQLite to the host machine, similar to the process for installing SQL Server and running a localhost instance. It seems that the .dll is all that's needed for code-first development of a database file. Thanks for pointing me in this direction.
    – LNX
    Oct 5, 2016 at 1:14

3 Answers 3

3

Is an object based on a text file approximately 50mb in size likely to be problematic if the app is running on a semi-modern Windows machine with 4gb-8gb of system memory?

Not the object itself. I'm more concerned with the way you plan on implementing the UI. The grid design you propose will require many, many times the amount of memory that holding the actual ADIF will require, and it won't exactly be user-friendly.

Am I correct that a hard-coded dictionary or enumeration does not occupy any memory space until it is called?

You can certainly make such objects lazy-loading. However, I've had a look at the enumerations, and the amount of memory required for them is almost certainly inconsequential anyway.

...without the use of a database solution such as SQLite or MS SQL

There are many compelling reasons why those other guys built their apps on top of a database system; you get loads of capabilities for free. Having all that log data in a database suddenly makes it useful for all sorts of purposes, not the least of which is filtering, searching, sorting and querying the resulting data.

3
  • Seconded. Also data integrity and consistency when your edits are applied. Oct 4, 2016 at 6:16
  • Searching, querying and sorting are all things I will need my app to do. I may need to just bite the bullet and incorporate a database into the app, but I believed I could do those things against an in-memory object and, if need be, write updates to a particular line containing a particular record in the text file.
    – LNX
    Oct 4, 2016 at 18:58
  • I forgot to mention paging, which is natively supported in System.Data.Sqlite.Linq using Skip() and Take(). Oct 4, 2016 at 19:09
2

I agree with the other answers here but I want to add a little detail as to why building this all in-memory and persisting to a file can be problematic.

If you were just building an application to view these files, then building a DB for it would probably be overkill. But you want to make edits and then save everything at the end. This approach is problematic:

  • If the application crashes (computer shuts down, user forces it down, etc.), all changes since the last time it was opened will be permanently lost.
  • If the application crashes during the save or has an bug in the save routine, you will corrupt the file and potentially lose the entire thing.

I've had the displeasure of working with applications that save changes on exit and I hate that about them. I consider such an approach a bad practice. To fix it with a file design, you need to, at the very least, back up the file before you attempt to write it. Then you'll need to be able to save the file either on change or let the user control when the save is done. Saving the entire file could be a little slow but the files are not very big so it might be OK. That's your minimal implementation.

If you want to make this better, though, you'd need to be able to update sections of the file without writing the entire thing every time. Once you get to this point of sophistication, using a database is much easier than trying to manipulate a file in this way.

3
  • Those are great points. I agree with your description of the minimal implementation. Writing each new record or edit to the text file as they occur is far better. To expand on my reason for not wanting to incorporate a database with this app, I was hoping to avoid the need for distributing and writing installers for whatever database is used. When the app is distributed it will need an installer, which would need to install SQLite or whatever I choose. Speaking of which, what is recommended for a WPF application that needs a purely local DB?
    – LNX
    Oct 4, 2016 at 19:00
  • @W4LKR You are not going to have row level access to text file to edit. That is what a database does. If you have good XML you can semi search.
    – paparazzo
    Oct 4, 2016 at 19:02
  • @W4LKR I'm not that familiar with the .NET ecosystem but it seems to me that an in-memory DB would fit your needs very well. In the Java world there is Derby, which effectively is just a library you include with the rest of your binaries. I'm not sure if there's an equivalent or port for .NET but it might be worth a search.
    – JimmyJames
    Oct 4, 2016 at 20:01
0

If you are talking .NET a single object or collection is limited to 2 GB and you don't get all of that as it needs to be contiguous. If it is only 50 MB on disk then you should be OK.

As Robert said - if your design is to display it ALL in one grid that is going to be more memory than the data. You may want to rethink that design.

You can certainly use collection and enumerations to store your data in memory.

You can search collections with LINQ.

I wrote small application where data was read from XML, manipulated in memory, and then written back to XML. But it would have been a lot easier to write it as a database application. Data in XML was a requirement.

1
  • I wouldn't need to display the entire collection in a grid. It would certainly be paged and only fetch 20 or 30 records at a time. Sorry for not stating that.
    – LNX
    Oct 4, 2016 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.