I read about microservices and it seems illogical to me to create a separate DB per service just to achieve isolation. I can achieve the same using only web services and a single database. Why do we even need it? The thing that separate database is beyond discussion. Or I am plain wrong? Can you guide me on this?
asked Oct 9, 2018 at 8:15
-
11databases are free– EwanOct 9, 2018 at 8:18
-
1One of the goal of microservices, among quite others, is to provide scaling beyond a monolitich architecture. Of course if your application won't even have the scale required for it, you might check your other requirment to see if it is worth to invest in microservices. Furthermore nothing prevent you do "dockerize" all or a part of your micro service in a same physical machine if you don't have the money to split them.– WalfratOct 9, 2018 at 9:18
-
1Services can easily share a database while remaining isolated: just give each service its own database user with access to its tables but not other services' tables.– Reinstate MonicaOct 12, 2018 at 17:53
-
Why do you need more than one code module? You can just put all the code in one big spaghetti class! It's less work!!! (The down side, of course, is that change management becomes a huge problem.)– John WuOct 12, 2018 at 20:51
-
@Solomonoff'sSecret that alone isn’t enough to isolate your services. One of those “users” could still execute a run away query that slows or takes everything down. It’s still a single point of failure. You’ve only logically isolated them.– RubberDuckOct 13, 2018 at 3:14
|
Show 2 more comments
5 Answers
Why do we even need it?
You don't.
Creating a separate database for each service helps to enforce domain boundaries, but it's only one approach. There's nothing stopping you from having all your services share the same database.
As long as your services behave and don't do unexpected things to data owned by other services, you'll be fine.
I don't know what you read, but you should be aware that there are many differing opinions on microservices architecture. Here's a good blog post on the topic.
I’ve seen folks refer to this idea in part, trivially, as “each microservice should own and control its own database and no two services should share a database.” The idea is sound: don’t share a single database across services because then you run into conflicts like competing read/write patterns, data-model conflicts, coordination challenges, etc.
But a single database does afford us a lot of safeties and conveniences: ACID transactions, single place to look, well understood (kinda?), one place to manage, etc.
The journey to microservices is just that: a journey. It will be different for each company. There are no hard and fast rules, only tradeoffs.
answered Oct 9, 2018 at 19:04
-
4In some environments, your storage is just another microservice anyway ...– svidgenOct 9, 2018 at 19:42
-
6You actually do need it. A major benefit of microservices is to be able to have a complete isolation of everything. A team may decide one day to move from a full Microsoft stack to LAMP, without even asking advice to other teams. If the same database is shared, you're not free any longer. Team A wants to move from SQL Server 2012 to SQL Server 2016, but team B cannot, because they are using a feature which was removed from the newer version. Unfortunately, this is not limited to versions; every thing two teams have in common may result in a conflict. Oct 12, 2018 at 18:06
-
2@ArseniMourzenko I understand that microservices should be platform agnostic and coupled only by service contracts, but it's not impossible to split a database shared by multiple services if you have a solid migration plan. In my previous role, I was the one arguing for separate databases for our multi-tenant healthcare application, but management chose a shared model due to cost concerns. I'm still frustrated by that over a year later. Oct 12, 2018 at 18:56
-
1I have also not seen an organization which actually allowed teams to use vastly different platforms (e.g. .NET vs LAMP). Such a rogue decision would be shot down pretty quickly over fears that certain services would end up in silos and could only be maintained by one team. Oct 12, 2018 at 18:56
-
@DanWilson: it is of course possible to split a database later. The problem is that when you start with a shared database, splitting becomes a difficult choice. Basic example: you would like a feature from the next version of the database; the other team cannot migrate yet. In most cases, you won't split (too difficult), but prefer not using the new feature, which is unfortunate. Oct 12, 2018 at 19:04
Why do we even need it?
The enormous benefit of microservices—and more largely, SOA—is the high level of abstraction of the internals—not only the implementation, but also the technologies being used. For instance, if a system is developed in a form of five microservices by five teams, one team can decide to move to a completely different technological stack (for instance from Microsoft stack to LAMP) without even asking other teams for their opinion.
Look at Amazon AWS or Twilio. Do you know if their services are implemented in Java or Ruby? Do they use Oracle or PostgreSQL or Cassandra or MongoDB? How many machines do they use? Do you even care about that; in other words, are those technological choices affecting the way you use those services?... And more importantly, if they move to a different database, would you have to change your client application accordingly?
Now, what happens if two services use the same database? Here are a tiny part of the issues which may arise:
The team developing service 1 wants to move from SQL Server 2012 to SQL Server 2016. However, the team 2 relies on a deprecated feature which was removed in SQL Server 2016.
Service 1 is a huge success. Hosting the database on two machines (master and failover) is not an option any longer. But scaling the cluster to multiple machines requires strategies such as sharding. Meanwhile, team 2 is happy with the current scale, and sees no reason to move to anything else.
Service 1 should move to UTF-8 as its default encoding. Service 2, however, is happy using Code Page 1252 Windows Latin 1.
Service 1 decides to add a user with a specific name. However, this user already exists, created a few months ago by the second team.
Service 1 needs a lot of different features. Service 2 is a highly critical component and needs to keep database features at their minimum to reduce the risk of attacks.
Service 1 requires 15 TB of disk space; the speed is not important, so ordinary hard disks are perfectly fine. Service 2 requires 50 GB at most, but needs to access it as fast as possible, meaning the data should be stored on an SSD.
...
Every little choice affects everyone. Every decision needs to be taken collaboratively, by people from every team. Compromises have to be made. Compare that to a complete freedom to do whatever you want in a context of SOA.
it's too [...] unmanageable.
Then you're doing it wrong. I suppose you're deploying manually.
This is not how things should be done. You need to automate the deployment of virtual machines (or Docker containers) which run the database. Once you automated them, deploying two servers or twenty servers or two thousand servers is not very different.
The magic thing about isolated databases is that it's extremely manageable. Have you tried managing a huge database used by dozens of teams? It's a nightmare. Every team has specific requests, and as soon as you touch something, it affects someone. With a database paired with an app, the scope becomes very narrow, meaning that there are much less things to think about.
If a huge database requires specialized system administrators, databases which are used by only one team can essentially be managed by this team (DevOps is also about that), freeing system administrators' time.
it's too costly
Define cost.
Licensing costs depend on the database. At the era of cloud computing, I'm pretty sure all major players redesigned their licensing to accommodate the context where instead of one huge database, there are lots of small ones. If not, you may consider moving to a different database. There are a lot of open source ones, by the way.
If you're talking about processing power, both virtual machines and containers are CPU-friendly, and I wouldn't be very affirmative that one huge database will consume less CPU than a lot of small ones doing the same job.
If your issue is the memory, then virtual machines are not a good choice for you. Containers are. You'll be able to span as many as you want, knowing that they won't consume more RAM than needed. While the total memory consumption will be higher for lots of small databases compared to a large single one, I suppose that the difference won't be too important. YMMV.
answered Oct 12, 2018 at 18:45
As Dan Wilson answers, you don't really need it. Microservices are the new hot thing, and like all new hot things, people use them in a lot of places even when they don't provide much value.
Microservices allow you to independently deploy and scale things at a "micro" level. That granularity provides a bunch of technical benefit and even more non-technical benefit since it allows you to better separate development teams, release as needed rather than one big release, try out new technologies or processes in isolation, etc. Having a shared DB kills a lot of that because of the dependency on the DB. If you can't deploy your service without worrying about other service's data, you've lost.
The thing that separate database is beyond discussion. Or I am plain wrong?
That said, you're also plain wrong.
When you're working in the cloud, databases are cheap. Free usually! Sure, the server costs money, but we're not talking about an individual server per microservice (at least, not at first). A single server with a bunch of (logical) databases is just fine as long as you're diligent about avoiding cross-database queries (which introduce dependencies that harm "independently deployable and scalable"). Hell, cross DB queries are impossible in some cloud database services like Azure SQL. You don't even need to be diligent there...
And I've even seen microservices where they shared a database, but each service got its own schema. Again, you need to be diligent about avoiding queries that cross data boundaries.
Lots of places aren't that diligent. They have entry level devs, or people who don't value the microservice approach, or have poor team leads, or have timeline pressure that causes people to take shortcuts.
Having a separate database is the cleanest way to enforce that decoupling that allows service independence, but it's not the only way. And it's not that expensive - especially when you compare it to the time/salary spent trying to enforce data boundaries in a shared database.
-
great. I guess if we are not the size of Amazon or uber then we should simply avoid it. Oct 10, 2018 at 6:36
-
1
-
In addition to all of the answers, from my understanding, putting a single database for most of your microservices is practical. It is useful to cover almost all database-per-microservice dilemmas (e.g., resource-draining for message passing, transactions, and ACID rules, eventual consistency over strong consistency, data redundancy, and maintenance) except:
- The fact that there should be a failure point for highly used tables. One or some tables of a module that lift heavy traffic is a failure point for the entire database.
- Some data may need to be kept in a DBMS other than the main DBMS (e.g., elastic for text data or new4j for graphs).
- Some teams need their autonomy over development, deployment, and even architectural design.
Hence, practically no need to provision a separate database for each and every service, but consider separating the ones
- that can be easily detached,
- or foreseen to be used heavily,
- or need to be stored in a different DBMS,
- or when a customer needs his/her data to be kept in his/her own place but you don’t want to hand over the whole database structure.
As a rule of thumb, use schema (or such) to logically group each table/collection by its service name so that they can be distinguished, learned, and separated more easily later.
Dependendin on what you consider “expensive”.
A database does not necessarily have to be an expensive commercial database server (think Oracle) not does it have to be necessarily a resource hungry affair. Depending on what are your requirements, you can use SQLite database or even file system as a persistent data storage.
All those services could also share a single database instance/server and only have isolated schemas per service.
The key argument here is that the service needs to own and control its data. How does it achieve that, is a matter of choice and technical details.
The best way a service can own and control its data is by having its own “personal” database. This allows for complete freedom of choice of the technology and data scheme evolution. The only way any other service can access data owned by a service is by asking for the data from a service. This way, if internal data representation needs to be changed, it can be easily changed and no other services will break.
So, to recap. It is not necessarily expensive to have a database per service nor is it necessary. It is simply a decision that you need to make at some point when developing microservices. Each of the choices has its implications and limitations. Study those and make your own choice.
