100

One of the major issues that I have seen occur in a system with microservices is the way transactions work when they span over different services. Within our own architecture, we have been using distributed transactions to resolve this, but they come with their own issues. Especially deadlocks have been a pain so far.

Another option seems to be some kind of custom-made transaction manager, which knows the flows within your system, and will take care of the rollbacks for you as a background process spanning over your entire system (so it will tell the other services to rollback and if they're down, notify them later on).

Is there another, accepted option? Both of these seem to have their disadvantages. The first one could cause deadlocks and a bunch of other issues, the second one could result in data inconsistency. Are there better options?

6
  • Just to be sure, are those microservices used by many clients at the same time, or just one at a time?
    – Marcel
    Jul 27, 2015 at 5:58
  • 2
    I was trying to ask this question agnostic to that, Marcel. But let's assume we are using a system big enough to do both of that, and we want to have an architecture that supports both.
    – Kristof
    Jul 27, 2015 at 6:02
  • 7
    This is a poorly worded, but very important question. When most people think about "transactions" they think almost exclusively about consistency and not the atomicity, isolation or durability aspects of transactions. The question should actually be stated "How do you create an ACID-compliant system given a microservices architecture. Only implementing consistency and not the rest of ACID is not really useful. Unless you like bad data. Jul 6, 2016 at 18:06
  • 14
    You can always try to alter my question to make it less "poorly worded", Jeff Fischer.
    – Kristof
    Jul 8, 2016 at 11:38
  • 1
    The most accepted transaction strategy for microservices is avoidance of transactions. Seriously. Mar 7, 2018 at 8:49

7 Answers 7

48

The usual approach is to isolate those microservices as much as possible - treat them as single units. Then transactions can be developed in context of the service as a whole (ie not part of usual DB transactions, though you can still have DB transactions internal to the service).

Think how transactions occur and what kind make sense for your services then, you can implement a rollback mechanism that un-does the original operation, or a 2-phase commit system that reserves the original operation until told to commit for real. Of course both these systems mean you're implementing your own, but then you're already implementing your microservices.

Financial services do this kind of thing all the time - if I want to move money from my bank to your bank, there is no single transaction like you'd have in a DB. You don't know what systems either bank is running, so must effectively treat each like your microservices. In this case, my bank would move my money from my account to a holding account and then tell your bank they have some money, if that send fails, my bank will refund my account with the money they tried to send.

3
  • 5
    You are assuming that refund can never fail. Nov 1, 2017 at 11:11
  • 11
    @SanjeevKumarDangi it's less likely and even if it does fail, it can be easily handled since the the holding account and the actual account is in the bank's control.
    – galdin
    Dec 27, 2017 at 10:29
  • There is some metadata associated, but n other services are one way to handle. Event sourcing is the other way to handle
    – MrMesees
    Jul 28, 2020 at 9:27
36

I think the standard wisdom is to never have transactions cross microservice boundaries. If any given set of data really needs to be atomically consistent with another, those two things belong together.

This is one of the reasons it is very hard to split a system into services until you have fully designed it. Which in the modern world probably means written it...

14
  • 44
    Such approach could very well lead to merging all microservices to a single monolithic application in the end. Mar 23, 2017 at 14:55
  • 8
    This is the correct approach. It is actually simple: if you need a transaction across services, your services are wrong: redesign them! @SlavaFominII what you say is only true if you don't know how to design a microservice system. If you find yourself fighting the microservices approach, don't do it, your monolith will be better than a bad microservice design. Only when you find the right service boundaries is when you should split the monolith in services. Otherwise using microservices is not the right architectural choice, it's just following the hype. Sep 2, 2018 at 15:49
  • @FrancescCastells What if our services genuinely require transaction among other services, do you mean that we should ignore bounded contexts and model our services in a way that it ends up as single transaction? I am a newbie in microservices, still reading so pardon my question if it sounds naive.... :D :D Sep 12, 2018 at 9:31
  • @BilboBaggins I mean the contrary of ignoring bounded contexts. By definition transactions happen within a bounded context and not across several of them. Therefore, if you correctly design your microservices along with your bounded contexts, then your transactions should not span multiple services. Note that, some time, what you need are not transactions, but better handling of eventual consistency and proper compensating actions when things don't go correctly. Sep 17, 2018 at 20:14
  • 1
    @BilboBaggins sorry, I completely missed your comments! Do you want to talk about this sometime? The question you propose is a good one. But the problem is not the answer, but the question itself. You say "there would be two different microservices" and you are probably thinking of a Customer and Account microservices and this is what is wrong. You need to abandon the idea of entity=microservice. This will allow you to see that you'll never need to store the name of the customer and his bank account in the same transaction, because there will never be a business rule that depend on this Oct 30, 2018 at 19:17
20

I think that if consistency is a strong requirement in your application you should ask yourself if microservices is the better approach. Like Martin Fowler says:

Microservices introduce eventual consistency issues because of their laudable insistence on decentralized data management. With a monolith, you can update a bunch of things together in a single transaction. Microservices require multiple resources to update, and distributed transactions are frowned upon (for good reason). So now, developers need to be aware of consistency issues, and figure out how to detect when things are out of sync before doing anything the code will regret.

But maybe in your case, you can sacrifice Consistency in pos of Availability

Business processes are often more tolerant of inconsistencies than you think because businesses often prize availability more.

However I also ask myself if there exists a strategy for distributed transactions in microservices, but maybe the cost is too high. I wanted to give you my two cents with the always excellent article of Martin Fowler and the CAP theorem.

3
  • 1
    In distributed business transactions, consistency is sometimes addressed by adding an orchestration layer like a workflow engine or a carefully designed queue. That layer handles all the two phase committing and rolling back, and lets the microservices focus on specific business logic. But back to CAP, investing in availability and consistency makes performance the victim. How do microservices compare to lots of decoupled classes comprising your business in OOP?
    – Greg
    Jul 27, 2015 at 14:02
  • You could use RAFT via microservices to address consistency FWIW
    – f0ster
    Jan 8, 2017 at 2:34
  • 1
    +1. I often wonder why in a microservices context transactions are desired at all, if all 'pull' views of the data can be implemented as materialized views. For example, if one microservice implements debits from one account and another credits to another account, the views of balances would only consider pairs of credits and debits, where unmatched credits and debits would still be in the buffers to the materialized view.
    – Sentinel
    May 23, 2018 at 13:09
16

As suggested in at least one of the answers here but also elsewhere on the web, it is possible to design one microservice which persists entities together within a normal transaction if you need consistency between the two entities.

But at the same time, you might well have the situation where the entities really do not belong in the same microservice, for example, sales records and order records (when you order something to fulfill the sale). In such cases, you may well require a way to ensure consistency between the two microservices.

Traditionally distributed transactions have been used and in my experience, they work well until they scale to a size where locking becomes an issue. You can relax the locking so that really only the relevant resources (e.g. the item being sold) are "locked" using a state change, but this is where it starts to get tricky because you are entering the territory where you need to build all the logic to do this, yourself, rather than having, say a database handle it for you.

I've worked with companies that have gone down the route of building their own transaction framework for handling this complex issue, but I don't recommend it because it is expensive and takes time to mature.

There are products out there which can be bolted on to your system which take care of consistency. A business process engine is a good example and they typically handle consistency eventually and by using compensation. Other products work in a similar way. You typically end up with a layer of software near the client(s), which deals with consistency and transactions and calls (micro)services to do the actual business processing. One such product is a generic JCA connector which can be used with Java EE solutions (for transparency: I'm the author). See http://blog.maxant.co.uk/pebble/2015/08/04/1438716480000.html for more details and a deeper discussion of the issues raised here.

Another way of handling transactions and consistency is to wrap a call to a microservice into a call to something transactional like a message queue. Take the sales record/order record example from above - you could simply let the sales microservice send a message to the order system, which is committed in the same transaction which writes the sale to the database. The result is an asynchronous solution which scales very well. Using technologies like web sockets you can even get around the problem of blocking which is often related to scaling up asynchronous solutions. For more ideas on patterns like this, see another of my articles: http://blog.maxant.co.uk/pebble/2015/08/11/1439322480000.html.

Whichever solution you end up choosing it is important to recognize that only a small part of your system will be writing things that need to be consistent - most access is likely to be read-only. For that reason, build the transaction management only into the relevant parts of the system, so that it can still scale well.

3
  • In the mean time, I'd say that one should seriously consider turning the process into an asynchronous one, where each step in the process is fully transactional. See here for details: blog.maxant.co.uk/pebble/2018/02/18/1518974314273.html Mar 10, 2018 at 9:59
  • "Make the sales record/order record example from above - you could simply let the sales microservice send a message to the order system, which is committed in the same transaction which writes the sale to the database." Not sure if what you are suggesting is basically a distributed transaction, that said, how would you handle rollback scenario in this case? E.g. message gets committed to message queue but got rolled back on DB side.
    – sactiw
    Mar 8, 2019 at 13:54
  • @sactiw not sure if I might have had two phase commit in mind, but I'd avoid that now and instead write my business data, and the fact that a message needs to be added to the queue, in one transaction to my microservice DB . The "fact" aka a "command" is then processed async after the transaction is committed, using an automated retry mechanism. See the blog article from 2018-03-10 for an example. Avoid rollback or compensation in favour of a forwards strategy if possible as it's easier to implement. Mar 9, 2019 at 16:20
2

There are many solutions that compromise more than I'm comfortable with. Granted, if your use case is complex, such as moving money between different banks, more pleasant alternatives may be impossible. But let's look at what we can do in the common scenario, where the use of microservices interferes with our would-be database transactions.

Option 1: Avoid the need for transactions if it all possible

Obvious and mentioned before, but ideal if we can manage it. Did the components actually belong in the same microservice? Or can we redesign the system(s) such that the transaction becomes unnecessary? Perhaps accepting non-transactionality is the most affordable sacrifice.

Option 2: Use a queue

If there is enough certainty that the other service will succeed at whatever we want it to do, we can call it via some form of queue. The queued item won't be picked up until later, but we can make sure that the item is queued.

For example, say that we want to insert an entity and send an e-mail, as a single transaction. Instead of calling the mail server, we queue the e-mail in a table.

Begin transaction
Insert entity
Insert e-mail
Commit transaction

A clear drawback is that multiple microservices will need access to the same table.

Option 3: Do the external work last, just before completing the transaction

This approach rests on the assumption that committing the transaction is very unlikely to fail.

Begin transaction
Insert entity
Insert another entity
Make external call
Commit transaction

If the queries fail, the external call has not taken place yet. If the external call fails, the transaction is never committed.

This approach comes with the limitations that we can only make one external call, and it must be done last (i.e. we cannot use its result in our queries).

Option 4: Create things in a pending state

As posted here, we can have multiple microservices create different components, each in a pending state, non-transactionally.

Any validation is performed, but nothing is created in a definitive state. After everything has been successfully created, each component is activated. Usually, this operation is so simple and the odds of something going wrong are so small, that we may even prefer to do the activation non-transactionally.

The greatest drawback is probably that we have to account for the existence of pending items. Any select query needs to consider whether to include pending data. Most should ignore it. And updates are another story altogether.

Option 5: Let the microservice share its query

None of the other options do it for you? Then let's get unorthodox.

Depending on the company, this one may be unacceptable. I'm aware. This is unorthodox. If it's not acceptable, go another route. But if this fits your situation, it solves the problem simply and powerfully. It might just be the most acceptable compromise.

There is a way to turn queries from multiple microservices into a simple, single database transaction.

Return the query, rather than executing it.

Begin transaction
Execute our own query
Make external call, receiving a query
Execute received query
Commit transaction

Network-wise, each microservice needs to be able to access each database. Keep this in mind, also in regard to future scaling.

If the databases involved in the transaction are on the same server, this will be a regular transaction. If they are on different servers, it will be a distributed transaction. The code is the same regardless.

We receive the query, including its connection type, its parameters, and its connection string. We can wrap it up in a neat executable Command class, keeping the flow readable: The microservice call results in a Command, which we execute, as part of our transaction.

The connection string is what the originating microservice gives us, so for all intents and purposes, the query is still considered to be executed by that microservice. We are just physically routing it through the client microservice. Does that make a difference? Well, it lets us put it in the same transaction with another query.

If the compromise is acceptable, this approach gives us the straightforward transactionality of a monolith application, in a microservice architecture.

2
  • 1
    About point 3: another drawback is that with external calls, you may not know whether the call succeeded or not (for example, a network failure); in this case, you may have your email sent and not know it; if you roll back the transaction, you end up with an email sent for a transaction that did not occcur. This is why I'm gearing towards point 2, committing a message as part of the transaction, that will be picked up by an async service. This way, you can split a business transaction into many independent database transactions, while ensuring that they will all be processed eventually.
    – BenMorel
    Dec 22, 2019 at 11:12
  • 1
    @Benjamin You're absolutely right. To have consistency guarantees, I also prefer this approach nowadays. And my caveat that "multiple microservices will need to access the same table" was not true: a microservice can simply queue things in its own table, and run its own background service to process them eventually (performing an idempotent operation to be able to guarantee processing, hopefully).
    – Timo
    Dec 23, 2019 at 12:25
1

In microservices there are three ways to achieve consistency between diff. services:

  1. Orchestration -- One process which manages the transaction and rollback across services.

  2. Choreography -- Service pass messages between each other and finally reach a consistent state.

  3. Hybrid -- Mixing the above two.

For complete reading go to the link : https://medium.com/capital-one-developers/microservices-when-to-react-vs-orchestrate-c6b18308a14c

0

I would start with decomposing problem space -- identifying your service boundaries. When done properly, you'd never need to have transactions across services.

Different services has their own data, behavior, motivational forces, government, business rules, etc. Good start is to list what high-level capabilities your enterprise has. For example, marketing, sales, accounting, support. Another starting point is organizational structure, but be aware that there is a caveat -- for some reasons (political, for example) it might be not the optimal business decomposition scheme. More strict approach is Value chain analysis. Remember, your services can include people as well, it's not strictly software. The services should communicate with each other via events.

The next step is to carve these services. As a result you get still relatively independent aggregates. They represent a unit of consistency. In other words, their internals should be consistent and ACID. Aggregates communicate with each other via events either.

If you think that your domain demands consistency first, think again. None of the big and mission-critical systems are built with this in mind. They all are distributed and eventually consistent. Check Pat Helland's classic paper.

Here are some practical tips about how to build a distributed system.

Not the answer you're looking for? Browse other questions tagged or ask your own question.