Several interdependent models, each with several data sources — how to avoid hell and nightmare

Question

I'm currently working on a project that requires a complicated model structure and I'm struggling with picking the right architecture.

First of all, there are several interdependent models. Change in one model can trigger changes in several other models. For example, user inputs a query, first model checks the database for type of the query and the second model requests its data depending on that query type.

Secondly, each model can have several data sources. Data sources are used conditionally, dependent on system configuration.

The app needs to function seamlessly, — for example, if one data source doesn't return required information, model must try and use another one.

An important thing also is that the codebase will eventually be expanded with other models. So models, data sources etc. must be very modular.

Currently there are several ways I can organise the model layer:

1. 'Dumb' models re-created by factory objects when input data (or a related model) changes. Model dependencies are resolved in an operation object, encapsulating the whole update routine. Operation object is configured and run by a domain model. I don't know where to inject a data source though.

2. 'Smart' models with update logic encapsulated in them. Each model queries a sort of a mapper/data-source-manager, which works as a proxy for different data sources. Models report their updates to a domain model. Dependencies are resolved by a model-controller or dependency manager object.

Which architecture is more appropriate here?

I'm leaning towards the first one as it is (to my mind) better to work with immutable models, updating the whole tree on change of input. This approach gives several benefits: for example it is easier to serialize as the presence of model equals to a finite state. But I'm really unsure about data sources...

Answer 1

"There are several interdependent models. Change in one model can trigger changes in several other models."

This is exactly what you should avoid. Welcome to Hell.

I'm getting a sense from your question that your head is in the technology more than in the business. I think the answer is for you to change your focus slightly...

The key to minimizing dependencies is in how you divide a business problem in order to solve it. It sounds like the existing design is already facing some challenges. This makes your job harder as you not only have to understand and interface with business needs, but also with legacy systems that do not currently meet those needs.

I would give the technical solution a rest and go back to the business and ask them 2 questions (again): What is the absolute minimum you need from the system on day 1? Where do you see the system capabilities going in the future? Any solution has limits and benefits. Communicate the relevant limits/benefits to the business (leaving out the technical details) and find out how well they fit with the business needs.

With the business, design for the future, but then pick a subset of that design to implement for the present. That way you keep it as simple as possible, without completely walling yourself off from the changes you are most likely to have to make to it. Another little trick I sometimes play is to imagine that the system the business and I just designed is all implemented and then imagine how it will be used with the business. Often this teases out critical design considerations that were overlooked. Use cases anyone? How the business will use the system is super-critical...

I don't know which of your proposed solutions you'll end up choosing, or if you'll come up with a new solution. No matter how smart I was, if I were to pick one for you, it would not be as helpful as for you to pick one in tandem with the business. The one that helps the business the most is the best way to go.

I hope that helps. 24 views, 7 hours, no answers or comments... I thought I'd give it a stab.

P.S. If you have to have fail-over from one data-source to another, that begs for a level of indirection somewhere. Ultimately, I'd want a layer in my code that I can ask for the data I need and if it's available at all, that layer will just return it. Then I can write relatively dumb clients that just say, "Get me the data" and not, "Get me the data using SQL-flavor A from Source #1... then if that doesn't work, try using another query language from source #2..." Maybe a JSON/XML/YAML service can encapsulate that complexity? Maybe it returns a little metadata footnote in addition to the data: "From source C because A took too long to respond and B gave an error."

I'm just latching onto the fail-over because it's the clearest requirement I got from your question.