Maintaining parallel libraries: Binary files access + Metadata Database ORM design

Question

I'm dealing with a higher-level data abstraction that I would appreciate some input on.

I'm working on an application that uses a large data lake. The data lake is consisted of thousands of large binary files (250MB+ up to many GB each), each of which contains a very small amount of metadata (a few kB) that uniquely identifies the file. The binary files ultimately contain the data of interest, but only a subset of the binary files are useful for a particular user's query. We created a database to act kind of like a "registry" for binary files. We built an ORM mapping to navigate the metadata effectively and efficiently to determine which binary files are useful for particular queries to tell which binary files actually need to be loaded. It may be relevant to know that users of this library won't have access to write, just read.

The design issue comes from the fact that now we are maintaining two parallel libraries for accessing data, a set of classes for data (including the metadata) inside the binary files, and the ORM classes for all the metadata. A good number of these classes are almost carbon copies of each other. Would it make sense to combine these libaries, and make the metadata objects always be orm entities? Keep in mind we want to keep it possible to load binary files without requiring database access. I'm hesitant to do this in part because it's such a drastic change but from a code maintenance perspective it would really clean things up.

We use sqlalchemy in python for the ORM if it matters.

edit: Here is a simplified example of the current state of the code base (sorry if this is too in-the-weeds).

current.py:

class BinaryMetadata:
    """Represents the metadata of each binary file. A few kB."""

    def __init__(*args, **kwargs)
        self.field_0 =  # ... save a few kB of
                        #
                        # ... metadata attributes
                        #
        self.field_n =  # ... in this file.


class BinaryFile:
    """A handler to read any part 1GB+ binary file, both 
    metadata and any data not in the database."""

    @classmethod
    def fromfilepath(cls, fp):
        """Load a file from disk"""
        with open(fp, 'rb') as file:
            args1 = # .... read metadata.
            args2 = # .... and any other data
        return cls(fp, BinaryMetadata(*args1), *args2)

    def __init__(self, fp, metadata, *args, **kwargs)
        self.filepath = filepath
        self.metadata = metadata # Instance of `BinaryMetadata`

    def read_binary_data(self, *args):
        # ...load 1GB+ of data binary file (not stored in ORM)...
        return binary_data

ORMBase =  #... base class for ORM


class ORMMetadata(ORMBase):
    """Mapping of table of full of data `BinaryMetatadata`"""

    id =  # ...
    field_0 =   # ... save the same  
                #
                # ... attributes here 
                #
    field_n =   # ... as in `BinaryMetadata`

The issue is that BinaryMetadata and ORMMetadata do damn near the same thing in terms of the data they store, and I'm wondering if it would be good design or terrible design to just use ORMMetadata in both places. It would sort of force api users to use the database api, but if we could merge the metadata classes and use just one in both places, we wouldn't have to maintain nearly identical classes ORMMetadata and BinaryMetadata.

Answer 1

Take a step back.

If there is something in common between the two API's then they are discussing the same thing.

You can extract that sameness into a library which is expanded on by the serialisation API, and by the Database API.

This will help in clarifying those common ideas, and also how those ideas are extended as you already have two examples.

Sharing structure creates rigidity

The only drawback with extracting the sameness is that it can become much less flexible. If the rigidity is in the right places this can help you to avoid mistakes, but if the rigidity is in the wrong places it can have some serious problems.

• Both API's rely on the shared library, any change to that library will impact both representations
• Both API's expose this shared library as part of their interface to downstream consumers. This forces the library to be imported regardless of consumer preference.
• If a consumer uses both API's, but those API's do not use the same version of the shared library, issues will occur. Technical binding issues are the small fish, the semantic issues will be worse.

Perspective

As both API's must create and interact with this common object model, it will be tough to make it flexible in the right ways. Perhaps changing your perspective can help clarify the problem.

Instead of considering this as two overlapping API's consider it instead a single API with two incarnations:

• One incarnation is Database Orientated, leveraging indecies etc...
• One is Flat File Orientated, leveraging a binary serialisation format.

From this perspective you can rightly see that the ERM and the Flat File are implementation details that could be hidden behind some sort of Provider interface.

In this sense they become Plugins to a single API:

• The consumer can choose to load no plugins, and gain access to the object structures themselves.
• The consumer could choose to load either or both plugins and gain access to the data stored in a file, or the lake.
• Tests can provide an in memory plugin which would allow for quick testing.
• The API can be updated to provide new features in a backward compatible manner.
• The plugins could be updated in such a way that they can be run in a backward compatible manner.