Designing access to file-based "database"

Question

It happened frequently that I have to provide access to a bunch of files organized in a directory tree according to some (sometimes loosely specified) rules. My standard pattern is to provide a Database class which is initialized with the root directory. This class provides getX()-like (example: getStructure()) methods to extract data from the database. These methods normally return semantically meaningful objects (Structure) with proper methods returning data (eg. structure.getPoints()). I am not completely happy with this design, for two main reasons.

The first problem is that the mapping between in-application objects and files may not be 1:1, that is, to create the Structure object I may have to open different files in the "database", and the mapping may not be perfect. In this case, I call the Structure a "Thick object". An alternative ("thin objects") is to stick to objects that are fully represented in a file, even if poor in high-level domain meaning (that is, if I have two files to define a structure, one file containing points, and the other containing connections between these points, I just provide a Points object and a Connections object, and let someone else "connect the dots" outside Database).

The other problem I have is the following: who should perform the actual file parsing? I envision two strategies: either the objects are able to deserialize themselves from file ("pickle-like" in python parlance), or they are just dumb data containers filled by stateless parser objects (one per file), or even by the database object itself.

When you do ORM, there are clear, well defined rules on how the object is represented in the database, and the process is extremely well defined in terms of interface and behavior. This is not necessarily verified with an arbitrary bunch of files some people call "database".

I would really enjoy your points on this regard. How should I perform proper deserialization in this case of "raw mapping"? Thin or thick objects ? Smart or dumb ?

Note that I don't really have any control at all when it comes to the files I have to access. I get the files from external sources, and I have to convert them into some kind of domain objects, generally only for reading, but sometimes also for writing. Readonly is by far the most frequent case though.

Answer 1

The key here is to have layers with well-defined interfaces. The lowest layer deals strictly with physical access to the files. It handles things like caching, concurrent access, opening, flushing, etc. Its interface allows higher layers to retrieve one line at a time without having to worry about things like if the file is already open.

The second layer provides dumb data structures to work with. Its interface would basically consist of a bunch of records with columns of simple data types, all of which are contained within a single file. Rows of the database, if you will.

The third layer provides smart objects to work with. It aggregates the database rows from multiple tables to make objects with friendly interfaces.

Answer 2

I call the Structure a "Thick object". An alternative ("thin objects") is to stick to objects that are fully represented in a file, even if poor in high-level domain meaning

These are not exclusive. You often need to do both.

You have physical format and logical layout issues separate from the problem domain. Three layers.

objects are able to deserialize themselves from file ("pickle-like" in python parlance),

Doesn't work out well in the long run. The class definition becomes cluttered with representation issues.

Python objects don't pickle themselves. Pickling is handled separately. Sometimes a class must provide codec hints to the pickle algorithm.

dumb data containers filled by stateless parser objects (one per file), or even by the database object itself.

"Stateless" parser objects don't always work out well.

Parser (or factory or builder) designs do work out well. Just leave off "stateless." Think of lexical scanning and parsing as separate from the physical, logical and final problem domain representations that may be required.

How should I perform proper deserialization in this case of "raw mapping"?

Physical Format classes may be necessary. Often, these are already provided in a language library. Don't reinvent the wheel.

Thin or thick objects ?

Not an exclusive choice. You have logical layout and the proper domain-level classes independent of any physical representation.

Smart or dumb ?

Outside the problem domain, classes should be focused on representation. I don't know of that makes them dumb or not.

Keep the concerns separate.

• Physical format is for CSV, ZIP, XML, JSON, etc.

• Logical layout handles variations in column names, data types, and structure.

• Problem domain is the real objects you wanted to work with.

Answer 3

I do this by serializing and deserializing objects using XML. This allows you to define your rules by how you structure your classes.

Some simple ToXml() and FromXML() helper methods do all of the heavy lifting.

So, for example:

public class MyRecord
{
    [XmlAttribute]
    public string Foo { get; set; }

    [XmlAttribute]
    public int Bar { get; set }

    [XmlAttribute]
    public double Baz { get; set; }
}

results in XML that looks more or less like this:

<MyRecord Foo="ABC" Bar="123" Baz="123.45">

You can make this as elaborate as you want, using composition to nest objects within other objects, and so forth, creating tree-like structures, or even collections.

public class MyRecordTree
{
    public MyRecordTree Node { get; set; }
    public List<MyRecordTree> Children { get; set; }
}