Over the last 20 years I've seen a few large modular database designs and I've seen the scenario suggested by David quite a few times now where applications have write access to their own schema/set of tables and read access to another schema/set of tables. Most often this data that an application/module gets read-only access to could be described as "master data".
In that time I have not seen the problems that prior answers are suggesting I should have seen so I think it is worth having a closer look at the points raised in the previous answers in more detail.
Scenario: you tie a couple of components to an RDBMS directly, and you see one particular component becoming a performance bottle-neck
I agree with this comment except this is also an argument for have a copy of the data locally for the microservice to read. That is, most mature databases support replication and so without any developer effort the "master data" can be physically replicated to the microservice database if that is desired or needed.
Some might recognise this in older guise as an "Enterprise database" replicating core tables to a "Departmental database". A point here is that generally it is good if a database does this for us with built in replication of changed data (deltas only, in binary form and at minimal cost to the source database).
Conversely, when our database choices do not allow this 'off the shelf' replication support then we can get into a situation where we want to push "master data" out to the microservice databases and this can result in a significant amount of developer effort and also be a substantially less efficient mechanism.
might want to denormalize the database, but you can't because all other components would be affected
To me this statement is just not correct. Denormalisation is an "additive" change and not a "breaking change" and no application should break due to denormalisation.
The only way this break an application is where application code uses something like "select * ..." and does not handle an extra column. To me that would be a bug in the application?
How can denormalisation break an application? Sounds like FUD to me.
Yes, the application now has a dependency on the database schema and the implication is that this ought to be a major problem. While adding any extra dependency is obviously not ideal my experiance is that a dependency on the database schema has not been a problem
so why might that be the case? Have I just been lucky?
The schema that we typically might want a microservice to have read-only access to is most commonly what I'd describe as "master data" for the enterprise. It has the core data that is essential to the enterprise.
Historically this means the schema we add the dependency on is
both mature and stable (somewhat fundamental to the enterprise and unchanging).
If 3 database designers go and design a normalised db schema they'll end up at the same design. Ok, there might be some 4NF/5NF variation but not much. What's more there are a series of questions that the designer can ask to validate the model so the designer can be confident that they got to 4NF (Am I too optimistic? Are people struggling getting to 4NF?).
update: By 4NF here I mean all tables in the schema got to their highest normal form up to 4NF (all tables got normalised appropriately up to 4NF).
I believe the normalisation design process is why database designers are generally comfortable with the idea of depending on a normalised database schema.
The process of normalisation gets the DB design to a known "correct" design and the variations from there ought to be denormalisation for performance.
- There can be variations based on DB types supported (JSON, ARRAY,
Geo type support etc)
- Some might argue for variation based on 4NF/5NF
- We exclude physical variation (because that doesn't matter)
- We restrict this to OLTP design and not DW design because
those are the schemas we want to grant read-only access to
If 3 programmers where given a design to implement (as code) the expectation would be for 3 different implementations (potentially very different).
To me there is potentially a question of "faith in normalisation".
Breaking schema changes?
Denormalisation, adding columns, alter columns for bigger storage, extending the design with new tables etc are all non-breaking changes and DB designers who got to 4th normal form will be confident of that.
Breaking changes are obviously possible by dropping columns/tables or making a breaking type change. Possible yes, but in practical terms I've not experienced any problems here at all. Perhaps because it is understood what breaking changes are and these have been well managed?
I'd be interested to hear cases of breaking schema changes in the context of shared read-only schema's.
What is more important and significant about a microservice: its API or its database schema? The API, because that is its contract with the rest of the world.
While I agree with this statement I think there is an important caveat that we might hear from an Enterprise Architect which is "Data lives forever". That is, while the API might be the most important thing the data is also rather important to the enterprise as a whole and it will be important for a very long time.
For example, once there is a requirement to populate the Data Warehouse for Business intelligence then the schema and CDC support become important from the business reporting perspective irrespective of the API.
Issues with API's?
Now if API's were perfect and easy all the points are moot as we'd always choose an API rather than have local read-only access. So the motivation for even considering local read-only access is that there might be some problems using API's that local access avoids.
What motivates people to desire local read-only access?
LinkedIn have an interesting presentation (from 2009) on the issue of optimising their API and why it is important to them at their scale. http://www.slideshare.net/linkedin/building-consistent-restful-apis-in-a-highperformance-environment
In short, once an API has to support many different use cases it can easily get into the situation where it supports one use case optimally and the rest rather poorly from a network perspective and database perspective.
If the API does not have the same sophistication as LinkedIn then you can easily get the scenarios where:
- The API fetches much more data than you need (wasteful)
- Chatty API's where you have to call the API many times
Yes, we can add caching to API's of course but ultimately the API call is a remote call and there are a series of optimisations available to developers when the data is local.
I suspect there is a set of people out there who might add it up as:
- Low cost replication of master data to microservice database (at no development cost and technically efficient)
- Faith in Normalisation and the resilience of applications to schema changes
- Ability to easily optimise every use case and potentially avoid chatty/wasteful/inefficient remote API calls
- Plus some other benefits in terms of constraints and coherent design
This answer has got way too long.