Microservices and data storage

Question

I'm considering moving a monolithic REST API to a microservice architecture, and I'm getting a bit confused about data storage. As I see it, some of the benefits of microservices would be:

• Horizontally scalable - I can run multiple redundant copies of a microservice to cope with load and/or a server going down.
• Loosely coupled - I can change internal implementations of microservices without having to change the others, and I can independently deploy and change them etc...

My problem is with data storage. As I see it there are several options:

1. A single Database service shared by all microservices - this would seem to completely eliminate any benefit of loose coupling.
2. A locally installed database instance on each microservice - I can't see a way of horizontally scaling this, so I don't think it would be an option.
3. Each microservice has it's own database service - this seems the most promising, as it preserves the benefits of loose coupling and horizontal scaling (using redundant database copies and/or sharding across several)

To me, the third option seems to be the only option, but it seems incredibly heavyweight to me, and a very overengineered solution. If I'm understanding it right, then for a simple application with 4-5 microservices I'd have to run 16-20 servers - two actual microservice instances per microservice (in case of server failure, and for deploying without downtime), and two database service instances per microservice (in case of server failure etc...).

This, quite frankly, seems slightly ridiculous. 16-20 servers to run a simple API, bearing in mind that a realistic project will probably have more than 4-5 services? Is there some fundamental concept that I'm missing that will explain this?

Some things that may help while answering:

• I'm the sole developer on this project, and will be for the foreseeable future.
• I'm using Node.js and MongoDB, but I'd be interested in language-agnostic answers - an answer might even be that I'm just using the wrong technologies!

Answer 1

Of your three options, the first (a single, shared database) and the third (a "database service") are the most common.

The first is called an integration database. This is generally not seen as a good solution in a microservice architecture. It does add coupling to your services. It also makes it very easy for one service to simply bypass the other services and query into a database directly. You could lose any kind of data integrity or validation provided by the application level not enforced at the database level.

Your third idea is called an application database. And you're right - it allows you to enforce the loose coupling at the API level between services and allows you to more easily scale services at the database level. It also makes it easier to replace the underlying database technology to something appropriate with each service, just as you can change technology or other implementation details of each service. Very flexible.

However, I'd propose an intermediate solution.

Instead of standing up a database service for every microservice, stand up a schema for every service. If you are using multiple database technologies, you may need to split slightly differently, but the idea would be to minimize the number of database servers that you are running, but make it very easy to split out a service into its own database server if and when it becomes necessary. As long as you only allow a database to access its own schema, you have the advantages of an application database but without the overhead of database servers existing for every application or service.

However, as a solo developer, I would challenge the entire notion of microservices at this point in time - Martin Fowler writes about the Monolith First and the Microservice Premium, Simon Brown talks about modular monoliths, and DHH talks about the Majestic Monolith. I'm not sure how well your monolith is organized, but refactor and organize it. Identify components and make clean separations between them for extracting pieces into a service easily. The same goes for your database structure. Focus on good, clean, component-based architecture that can support refactoring into services. Microservices add a lot of overhead for a single developer to build and support in operations. However, once you actually have a need to scale part of the system, use your monitoring and reporting systems to identify the bottlenecks, extract to a service, and scale as necessary.

Answer 2

Each microservice has it's own database service - this seems the most promising, as it preserves the benefits of loose coupling and horizontal scaling (using redundant database copies and/or sharding across several)

Agree. The third option is the natural choice for micro services. If the micro service are intended to be really independent (and not part of a distributed monolith), it's normal that they have, each one, a database.

[...] two actual microservice instances per microservice (in case of server failure, and for deploying without downtime), and two database service instances per microservice (in case of server failure etc...).

You are right about the quantity of micro services running if you would like to have a load balance. If you are planning to have 4 micro services you need to prepare at least 2 instances of each micro service (8 in total), as you already explain.

But two databases per micro service? This is really questionable. I don't know the details about the business problem that your micro services will attend, but have a database redundancy it's quite a lot for the most of the products/projects. I will recommend to start with one single database with a good backup and minimize (at least initially) the complexity of your infrastructure.

This, quite frankly, seems slightly ridiculous. 16-20 servers to run a simple API, bearing in mind that a realistic project will probably have more than 4-5 services? Is there some fundamental concept that I'm missing that will explain this?

For a simple API this numbers not match. Pay attention if you are not falling into one of the "Microservice First" traps.

Answer 3

Microservices are a form of Service Oriented Architecture, perhaps in the extreme.  Their general purpose is to reduce coupling and allow for independent development & deployment.

Very roughly speaking in architectural terms, microservices is a term that applies, let's say, at a logical level.  The microservices are logically separated from each other.  From this perspective microservices should each own and provide for their own storage, which should be decoupled from the storage of other microservices.  For microservices, this independence of storage is key to their goal of modularity and loose coupling.

From an architectural perspective, horizontal scaling applies at a lower level, closer to the implementation, let's say, at a physical level.  At this level, we're implementing a microservice, and we can decompose this single microservice, internally, into a stateless component that is horizontally scalable, and a stateful component that is shared by all the stateless components.  But let's not confuse just the stateless part alone with the microservice itself.

So, when we're talking about the individual microservices, we're at the logical level speaking about APIs and separated responsibilities and separated development/deployment cycles. And when we're talking about horizontal scaling, we're at the physical level talking about implementation of a (single) microservice and its decomposition into stateless and stateful components.

When implementing multiple microservices, we have choices for reusing database technology for the stateful components:

• separate database per microservice
• shared database with:
  • separate/private schema per microservice
  • separate/private tables per microservice

See more here.

A single Database service shared by all microservices - this would seem to completely eliminate any benefit of loose coupling.

Agreed, if you mean sharing tables rows & columns, that would not really be microservices.

If we can decouple — in our thought processes — the logical notion of microservices from the more physical notion of stateful and stateless components of a microservice, we may find it easier to achieve the loose coupling offered by microservices, while retaining the efficiency of a shared databases.

Generally, there's a fair amount written up about microservices and stateful persistence, see also here.

Answer 4

Well, I've read through all posts in this thread and can tell you that I am confused with the question: it mixes microservice (MS) with services with data access service (DB service) with databases and servers...

If MS is an independent (deployable) component solving one simplest task in a stateless manner, WHAT database it needs? If a more complex task to be solved, which requires more than one simplest sub-tasks (MS?) to be resolved together, is it a still a MS? In SOA, it is called an Orchestrating Service. It implements "process" and coordinates MS invocation, thus it needs to persist its state (all orchestrations/organisers/composers/etc. are stateful) and needs a personal datastore: nobody else may temper the state of the orchestrator.

However, we talk not about database but about Database Access MS/Service, and this is a totally different matter. An MS might need some data collected in the company (not in the Application it operates within) and it cannot ask another MS via its API/interface for the data. This is the most common and realistic scenario. And another MS from the same or different Application might need this data or even changes it. Yes, they compete for the data as it was always before MS emerged.

Why we are crash the 'door', which is well known and open? What is the difference with this regard between an MS and a regular self-persisted object? Why we need an individual database for MS if it must (for the flexibility and composability purposes) engage its Data Access Service (DAS) anyway? Do not forget that DAS shields the MS with a business task from the awareness about physical connectivity to the database. This loose-coupling and flexibility the MS should preserve to freely participate in multiple Applications with no database anchor on it.