Rebuilding a legacy application : What about the data model?

Question

I am currently rebuilding from scratch a product catalogue (to feed a shopping website). The existing legacy system is heterogenous, Long story short : The some products are stored in a FileMaker database (with a big spaghetti data model) and other products are edited as YAML files ...

So my question is : what are the strategies for data modeling if I want to go step by step migrating to the new system, that is to say, do I have to make an exhaustive data model as soon as possible (different kinds of products, handle taxonomy, products variants, stock management, ....) or can I begin small, only modeling a particular kind of product and growing my data model as I'm adding more products types ?

The advantage of begin small is that I don't have to take account of all details at the beginning (do something that work - minimum viable product) but in the other hand it can mean that the choices made earlier can stuck me in the long run.

What would be the most effective approach?

Answer 1

Think big, start small.

The most effective way I know for tackling the described situation is the following approach:

• Scetch your "vision" of the full data model, but do not implement it with all the gory details in some database. Instead, keep the model in a conceptual form. You can do this on a whiteboard, a piece of paper, some UML drawing tool, a word processor, or whatever you prefer. It should be in a form where you can easily and quickly apply changes in a few minutes, whenever necessary.

• When going to implement the system, do this incrementally in small steps. Only flesh out the parts of the model in full detail for which you are also implementing some business processes using these parts of the model.

The conceptual "vision" will help you to prevent painting yourself in a corner and to stay focussed on your goals. However, it will not stop you from providing a small, working product soon. The implementation of those business processes will help you to verify whether the decisions you made for the data model were correct, allowing you to fix the decisions in case they reveal defects, and will also give you some feedback if the general concept is still sensible.

Let me add that some years in the past I worked in an enterprise project where the task was similar to yours: replacing an old legacy system (mainframe) by a new system based on some Oracle enterprise database. The business analysts started with modeling all the attributes they found in the old system long before any real business code was written. After some years, the whole project ended in a desaster. There were several reasons for this, maybe the data model was not the primary reason. But more than half of the attributes turned out to be unsuitable, either under- or overengineered, and the whole thing and development organization around proved itself to be way-too-unflexible for allowing to rescue the project.

Answer 2

My approach to this problem would be, ( I am bit simple, step by step in terms of design and coding )

1. If you have enough time, spend more on design phase.
2. Study the schema of the xml files and the file maker databases.
3. List down all the individual properties and object orientation would be good to adhere here.
4. Then spend time to see if they have overlapping properties, one good place to find this out is to check - how they are rendered in the UI. If they are displayed in the same row - during a product listing - it might imply they are similar data but might be called differently. For example, T shirt size in xml and size in database.
5. Build code to assimilate and transform the individual entries into a new hybrid model may be in XML.
6. Build a model tester which tests data in xml and db against the newly generated data - to catch flaws or gaps.
7. You can rerun this model tester often after doing changes, to make sure all the data is fully transformed.

I worked on something similar to put together affiliate feeds from different vendors.

Answer 3

(Koff, koff ...) "Sometimes, the 'data models' that we had to use were simply those that the product-de-jour handed to us at that time. For instance, if "the final deliverable" was "a report," then we had to feed it. Likewise: "your situation."

At This Point: you are dealing with "re-implementation." Virtually nothing of "what you are now presented with" will actually "carry forward." And, many of the data representations, most of them being "the necessities of their technical times," will also not carry forward. Instead, they are merely "sources of data." And, the technologies and techniques that "they, then, supported," are merely suggestions now.