9

I’m finding it hard to avoid data duplication or a shared database for even the simplest microservices design, which makes me think I’m missing something. Here’s a basic example of the problem I’m facing. Assuming someone is using a web application to manage an inventory they would need two services; one for the inventory managing the items and the quantity in stock and a users service that would manage the users data. If we want an audit of who stocked the database we could add the users ID to the database for the inventory service as a last stocked by value.

Using the application we may want to see all the items that are running low, and a list of who stocked them last time so we can ask them to restock it again. Using the architecture described above, a request would be made to the inventory service to retrieve the item details of all items where the quantity is less than 5. This would return a list including the user IDs. Then a separate request would be made to the users service to get the user name and contact details for the list of user IDs obtained from the inventory service.

This seems awfully inefficient and it doesn’t take many more services before we’re making multiple requests to different services APIs which in turn are making multiple database queries. An alternative is to replicate the users details in the inventory data. When a user changes their contact details we would then need to replicate the change through all other services. But this doesn’t seem to fit with the bounded context idea of microservices. We could also use a single database and share this between different services, and have all the problems of an integration database.

What’s the correct/best way to implement this?

  • 4
    Welcome to the paradox of micro-services. That which would appear to make things simpler can actually makes things more complex. – Robert Harvey Mar 19 '18 at 15:04
  • The "correct" way is the same as it's always been: figure out a way of doing things that best suits your specific objectives. – Robert Harvey Mar 19 '18 at 15:05
  • @RobertHarvey That's always the case but I'm trying to understand the textbook microservices way. Once I understand how it should work in an ideal world I'll happily change it to fit my use case. – Geraint Anderson Mar 19 '18 at 15:29
  • But your framing your question in terms of efficiency, which is a non-functional software requirement. The way you solve the efficiency problem is by asking the database directly. – Robert Harvey Mar 19 '18 at 15:36
  • I was about to write a question exactly as yours.I still don't see advantages in MSA for reasonably simple web applications. I think in many cases modularity could be achieved without making things so complex. – Glasnhost Oct 26 '18 at 17:49
5

I completely missed where you're being required to duplicate.

A central principle of micro services is for the service to be the single authority. That means inventory and user management can be completely separate. I'd design the user management so that it doesn't even know the inventory system exists.

But I'd design the inventory system so that it never stores anything about users other then a user ID. That takes care of your problem of propagating user info changes.

As for things that need both inventory info and user info such as logs, audits, and print outs they don't get updated as info changes. They are a record of what was. Again, you don't propagate change.

So in every case, when you want the latest user info you ask the user info service.

  • @Geraint: Can you be more specific about what kind of duplication is occurring in your system? – Robert Harvey Mar 19 '18 at 15:38
  • 1
    Thanks. The duplication referred to copying the users contact details to the inventory service but you have addressed that (i.e. it's not required). It seems counter-intuitive to move from a single relational database where I could get the inventory data and the user data with a join to making two distinct API calls where the second can't begin until the first has returned the results. But I guess that's part of the evaluation as to whether I use microservices or something else. – Geraint Anderson Mar 19 '18 at 16:03
  • It's the same trick the DB would use if it managed both. You don't copy user info into the inventory table. You give it a foreign key. The user ID is doing the same job across services. Just make it unique. – candied_orange Mar 19 '18 at 16:06
  • It seems counter-intuitive to move from a single relational database where I could get the inventory data and the user data with a join Keep in mind that "ideally" there's one store per service (or more!). So, there's nothing such as "join" between "boundaries". The reason is simple, DB generates coupling among services. Unlike @CandiedOrange suggest, I think we can duplicate a minimum of data from one service to another. I'm referring to data which is unlikely to change. If this dups improves efficiency and performance (and both are required) the "pros" would probably off-set the "cons" – Laiv Mar 19 '18 at 16:17
  • @GeraintAnderson I mean, if you need efficiency (which is by definition a non-functional requirement), there are ways to do that. I.e. request pages of data from the Inventory Service (like 10 elements), take each page and use that page to request data from the User Service, and aggregate at the end. That way you keep your boundaries while leveraging the parallelism of independent services. Even then, don't bother until you've identified it as a real bottleneck of the application that must be resolved - waiting an extra 1/2 second on a 1-second overnight job doesn't matter to anyone. – Delioth Jan 8 at 21:14
6

I’m finding it hard to avoid data duplication....

According to the Microsoft ebook on microservice architecture, there is nothing wrong with data duplication. Basically, duplicating data increases the decoupling between the services and therefore strengthens their roles as a single authority. A relevant passage:

And finally (and this is where most of the issues arise when building microservices), if your initial microservice needs data that's originally owned by other microservices, do not rely on making synchronous requests for that data. Instead, replicate or propagate that data (only the attributes you need) into the initial service's database by using eventual consistency (typically by using integration events...

3

a request would be made to the inventory service to retrieve the item details of all items where the quantity is less than 5. This would return a list including the user IDs. Then a separate request would be made to the users service to get the user name and contact details for the list of user IDs obtained from the inventory service.

Indeed, yes.

Granted, in a monolith you could have an Inventory-model that you query for the relevant items, feed that into a User-model and get the same data.

Or you could take it further, if you have them in the same relational database and write SQL that and the database will take the inventory-table and user-table, it does some magic, and you get the data you are after.

Regardless of how you do it, somewhere there will be code that essentially fetches a list of user ids from the inventory system, feeds them into the user system and compiles a list of data.

The question you need to answer is about performance and maintenance and other "soft" qualities.

The main benefit of microservices is scaling. If you have a ten thousand users on one machine and it is a bit sluggish, you can add another machine and the system becomes twice as fast. Add eight more and it's ten times as fast. (Linear scaling is probably optimistic, but it is the ideal and not that unreasonable to hope for.)

And this is per service. If the inventory system is the bottleneck, it is used for more than reports about users, you can add more machines to just that service. The machines can also be specialised; this service needs a lot of memory, that service does heavy calculations and needs more cpu.

If you don't need the scaling, there is one other benefit of microservices: they are modular. Of course, monolithic apps can also be modular, and you have a normalised database and... but in practice the walls between modules are like glass walls in the best case, and lines in the sand in the worst. Microservices are separated by solid steel.

If your user system literally catches fire, that wont affect your inventory system in the slightest. You wont be able to print pretty reports about who stocked what, but customers will be able to place orders safe in the knowledge that the stocked items are there.

And you don't duplicate data in microservices, any more than you do in a relational database(*). In a relational database you can do a join, and the equivalent is to merge the lists in code like described.

You could also add a view, the equivalent is to add a new service that does the merge for you; that would result in three requests; one to the new service and then that service does the original two. Relational databases have fancy stuff that optimises views, that has to be implemented on the service level. You don't get it "for free".

Caching is different from data duplication in that if two values mismatch you know which one is wrong. It is often used in microservices to bring availability up at the expense of consistency (CAP theorem). Since relational databases completely butcher availability on the altar of consistency it is less common in them. I'd say there is nothing inherent about microservices that makes caching easier, but in practice caching is a primary concern and that makes caching easier in microservices.

(*) If it makes sense to duplicate data in a microservice swarm then it probably would make sense in the equivalent relational database to.

  • 2
    I really liked your answer until the "don't duplicate data in microservices" part. I think there are cases where data duplication is the right approach. It improves fault tolerance and autonomy. If the user service went down, the inventory service can still display a list of low inventory with who stocked them last. – Peter Pompeii Feb 16 at 19:35
  • @peterpompeii I'd call that caching, not data duplication. Data duplication is when you have two place to update for one datum, caching when there is one place and automatic propagation to the other places. Also I said more than relational. If it makes sense in a relational database to duplicate data it makes sense in a microservice. I think we agree and that part could be clearer, but I only have a phone right now so won't update the text right now. – Odalrick Feb 18 at 1:00
  • @PeterPompeii Hope the added section about caching addresses some of your concerns. – Odalrick Mar 6 at 7:28
  • @Odalrick what you described sounds like data replication. Replication and caching are both forms of duplicating data. Replication is when a copy is guaranteed to always have all the needed data. Caching is on-demand. Caching can have a miss. Caching for availability does not make as much sense as caching for performance. TL;DR if you are storing a complete copy of something with enough consistency guarantees that you never need to check for misses, then it's not a cache. – Brandon Apr 25 at 21:04
  • @Brandon I see your point, but it's not what I was thinking of. I'm talking about caching the result of calculations, like map-reduce in CouchDb. You cannot recreate the full data from them. Replication is almost something you do instead of microservices; get a big fat database instead of splitting the database into less coupled services. – Odalrick Apr 26 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.