Practical way to store a "reasonably large" amount of data that hardly ever changes?

Question

Think in terms of pre-computed lookup tables or something. At what point does it make more sense to use a database instead of hardcoding values in my application? The values are not going to change, and they're nicely segregated away from maintenance developers. 100 values, 1k, 10k, 100k? I'm wanting to store about 40k values. Right now it's a machine-generated switch statement (which VS2010 is unhappy about).

edit:

If anyone is curious, here's how I approached this: My data was storable in two 100k-element float arrays, so that's what I did. It took about 20 seconds to generate the data, so I did that once, and serialized it to an embedded resource with a BinaryFormatter. Unpacking the data takes about 5 milliseconds at application startup, and outperforms the database implementation I was replacing (these hard-coded values were stored there prior) by almost 45,000x.

Answer 1

My suggestion is to keep the data in either a file or database table. If speed is not an issue, then query the file or database (database is better) at run time. If memory is not a problem, but you would like some speed, then load the data into memory when the program starts. In C# you could use and array, list or (best option) a hash-table and have a method to return the data you need at runtime ( ie getDataValue(string keyToValue) ).

I would recommend that you do not use the switch statement as it would be very difficult to maintain and would result in a large exe footprint.

Hash-table eg http://support.microsoft.com/kb/309357

Answer 2

Personally, I am Ok to store any amount of data, hardcoded into the application, until there is no need to tweak it for one particular deployment or hotfix.

However, storing and accessing data using C# switch statement, is rather bad practice, since in tightly couples data storage and data access model and implies only one method access method (by switch parameter).

I would prefer to store data in a Hashtable or Dictionary, and provide separate classes for retrieving the data, and one time populating of look-up Dictionaries.

Recently, I've found rather convenient to implement small DSL for specifying business rules ( fluent interface for SiteMap or tax calculator interview question check "calc" method for rules defenition ) and then provide separate object for querying these rules. This technique would apply well for switch case scenario.

One of the nice benefits of such decomposition, is that you can implement a number of Views on your data, without touching XXXk lines blob, that defines that data.

Answer 3

A 40k line switch statement is a little questionable. I assume you still need to perform query operations right? Have you tried encapsulating the data? Then use LINQ to perform query operations on the collection to test performance. Get some concrete times by running unit tests with a timer like StopWatch. Then, if you think it might just work. See if performance is acceptable for the users.

Answer 4

I've had a requirement like this twice. The applications were designed to be standalone with no database setup/access required. In both cases I used XML files to store the data. In the first one, which was on the 2.0 Framework, I used the old style XML parsing calls to look up data. For the newer one, on the 3.5 Framework, I used LINQ to XML to find what I needed. In both cases, access to the data was encapsulated in classes.

Answer 5

The key thing here is to make sure your public interface encapsulates your implementation- but that's not your question and there is no reason to think you haven't. Beyond that, it's just a question of performance vs grief (and performance differences may well not be worth caring about). As a practical solution, for the VS 2010 issue, you could always break the case statement up into a hierarchy of case statements - the top level could call one of 10 other methods, each with a case statement of 4000 cases, for example. You could put each of the 10 into it's own file if you had to. A little ugly, but you are code generating anyway.

As for the number to switch to a DB -it's just whenever not using a DB becomes a problem.

Answer 6

You could use something like SQL Compact. Put the data in a table and leave the DB file in the project. Tables are better suited for that amount of data than a switch statement.

Answer 7

I think the key word here is 'hardly'

If the data never changes - for example, pre-calculated mathamatical values, color constants, and the like - then sure, as long as the size is managable for you, keep it in the code. Just be aware that if performance is an issue, case/switch statements will be very slow compared to other options.

If the data hardly ever changes - for example, phone area codes, national boundaries, and the like - I'd probably look into keeping the data externally in some manner. Particularly if it started getting to be more than a couple dozen values.

Answer 8

If you store large volumes of data into your application, then your program may load slower and you may be exposing the code to risk in case some one could play with the binaries or executable.

Also, if the program is edited many times, who knows, may be you could introduce errors by mistyping a numbers by mistake or as a result of change command.

May be in the future some one asks for running queries on the data, say, someone may ask for the average, of a column, in which case you will have to change your application and add a method to calculate every query your user comes up with, then go with the all the steps to promote your code to production. This is really not good.

Separating data and code is a good practice specially if the data is large.