1

Imagine this use case:

I have a class with 50 attributes of which 10 are relationships plus 100 methods which perform calculations and return a value.

I need to save that data (including relationships and method's return values) into a database for later retrieval. The idea is to have an snapshot of the current state of the object instance.

Currently I only need the data of 20 of that attributes and 25 method return values, but in the future will need to retrieve more. I will probably never use some of the data saved.

I could easily save and retrieve data from a JSON.

Now imagine that after 3 months having generated and saved 1.000.000 instances of this class, I need to start changing things eventually, for instance:

  • Change the name of 2 attributes and 3 methods
  • Drop one relationship
  • Split the value of an attribute into 2 parts (for instance object.name into object.first_name object.last_name)
  • Add 5 new methods and attributes

While I need to not break the data retrieval for historical data (That is, update registries or process accordingly on fetching).

Having considered that use-case:

What's the best approach for data retention, manipulation and retrieval? (speaking in terms of databases, design patterns and other technical means)

Improve this question
4
  • We could answer your question better if you provided more specifics about the nature of the data. What kind of data are you storing? What are the tables and fields? Why doesn't an ordinary database design suffice?
    – Robert Harvey
    Commented Jan 31, 2018 at 15:54
  • Have a look at EAV. It solves the problem of "many fields but few stored," though it comes with its own costs. en.wikipedia.org/wiki/…. Your scenario might also be one of the few exceptions where a NoSQL/Document database is preferable to an RDBMS.
    – Robert Harvey
    Commented Jan 31, 2018 at 17:03
  • Many fields stored but few retrieved is not a problem that needs to be solved. Last time I checked, hard drive space cost about ten bucks per terabyte. If some fields will never be retrieved, they don't have to be stored at all.
    – Robert Harvey
    Commented Feb 1, 2018 at 4:07
  • Let us continue this discussion in chat.
    – Hacktivista
    Commented Feb 1, 2018 at 14:19

2 Answers 2

Reset to default
1

I would implement object-serilisation and object-deserilisation via human readable textformat (xml/json/csv) for all businessclasses.

Example:

  • every class gets a toString() method that creates a string of the object and it-s subobjects
  • every class get a fromString(string) that recreates the object with it-s sub-objects. This method must understand the current format and all old formats.

This aproach is database independant.

Datamanipulation happens completely in the objectmodell.

Improve this answer
3
  • There's a lot of costs to this approach, so the benefits better be worth it.
    – Robert Harvey
    Commented Jan 31, 2018 at 15:55
  • Seems like that could be a lot of understanding between formats.
    – JeffO
    Commented Jan 31, 2018 at 18:34
  • Will use this approach... We were capturing data in a JSON already, and the idea of recreating exactly the needed data using its models is appealing :) I'll use versioning in the JSON too in order to know how to process the input if there's more than one change in the field or related fields. In any case this approach has some quirks, as not being able to process large quantities of data in a timely fashion. But we can manage it by using other techniques. In any case I think the problem is very interesting, and other interesting options are plausible, so please keep answering!
    – Hacktivista
    Commented Feb 1, 2018 at 14:35
1

What you're asking about is database refactoring, and if you search for that you'll find a lot of material as this is ground others have had to cover for various reasons.

Some have the challenge that their software is supporting multiple clients each with their own installation (of the database), meaning that any latest version has to be able to upgrade from any existing version to the current version. Needless to say, automation is important keyword to add here.

Evolutionary Database Design by Martin Fowler, for example, goes through some examples that you mention, and discusses the importance of versioning with respect to data, which means using version control, version numbers so you can recognize the data, etc..

Improve this answer
1
  • This does not solve the problem of more data saved than used.
    – Hacktivista
    Commented Jan 31, 2018 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.