I'm dealing with a higher-level data abstraction that I would appreciate some input on.
I'm working on an application that uses a large data lake. The data lake is consisted of thousands of large binary files (250MB+ up to many GB each), each of which contains a very small amount of metadata (a few kB) that uniquely identifies the file. The binary files ultimately contain the data of interest, but only a subset of the binary files are useful for a particular user's query. We created a database to act kind of like a "registry" for binary files. We built an ORM mapping to navigate the metadata effectively and efficiently to determine which binary files are useful for particular queries to tell which binary files actually need to be loaded. It may be relevant to know that users of this library won't have access to write, just read.
The design issue comes from the fact that now we are maintaining two parallel libraries for accessing data, a set of classes for data (including the metadata) inside the binary files, and the ORM classes for all the metadata. A good number of these classes are almost carbon copies of each other. Would it make sense to combine these libaries, and make the metadata objects always be orm entities? Keep in mind we want to keep it possible to load binary files without requiring database access. I'm hesitant to do this in part because it's such a drastic change but from a code maintenance perspective it would really clean things up.
We use sqlalchemy in python for the ORM if it matters.
edit: Here is a simplified example of the current state of the code base (sorry if this is too in-the-weeds).
class BinaryMetadata: """Represents the metadata of each binary file. A few kB.""" def __init__(*args, **kwargs) self.field_0 = # ... save a few kB of # # ... metadata attributes # self.field_n = # ... in this file. class BinaryFile: """A handler to read any part 1GB+ binary file, both metadata and any data not in the database.""" @classmethod def fromfilepath(cls, fp): """Load a file from disk""" with open(fp, 'rb') as file: args1 = # .... read metadata. args2 = # .... and any other data return cls(fp, BinaryMetadata(*args1), *args2) def __init__(self, fp, metadata, *args, **kwargs) self.filepath = filepath self.metadata = metadata # Instance of `BinaryMetadata` def read_binary_data(self, *args): # ...load 1GB+ of data binary file (not stored in ORM)... return binary_data ORMBase = #... base class for ORM class ORMMetadata(ORMBase): """Mapping of table of full of data `BinaryMetatadata`""" id = # ... field_0 = # ... save the same # # ... attributes here # field_n = # ... as in `BinaryMetadata`
The issue is that
ORMMetadata do damn near the same thing in terms of the data they store, and I'm wondering if it would be good design or terrible design to just use
ORMMetadata in both places. It would sort of force api users to use the database api, but if we could merge the metadata classes and use just one in both places, we wouldn't have to maintain nearly identical classes