One of the major issues that I have seen occur in a system with microservices is the way transactions work when they span over different services. Within our own architecture, we have been using distributed transactions to resolve this, but they come with their own issues. Especially deadlocks have been a pain so far.

Another option seems to be some kind of custom-made transaction manager, which knows the flows within your system, and will take care of the rollbacks for you as a background process spanning over your entire system (so it will tell the other services to rollback and if they're down, notify them later on).

Is there another, accepted option? Both of these seem to have their disadvantages. The first one could cause deadlocks and a bunch of other issues, the second one could result in data inconsistency. Are there better options?

  • Just to be sure, are those microservices used by many clients at the same time, or just one at a time? – Marcel Jul 27 '15 at 5:58
  • 2
    I was trying to ask this question agnostic to that, Marcel. But let's assume we are using a system big enough to do both of that, and we want to have an architecture that supports both. – Kristof Jul 27 '15 at 6:02
  • 3
    This is a poorly worded, but very important question. When most people think about "transactions" they think almost exclusively about consistency and not the atomicity, isolation or durability aspects of transactions. The question should actually be stated "How do you create an ACID-compliant system given a microservices architecture. Only implementing consistency and not the rest of ACID is not really useful. Unless you like bad data. – Jeff Fischer Jul 6 '16 at 18:06
  • 4
    You can always try to alter my question to make it less "poorly worded", Jeff Fischer. – Kristof Jul 8 '16 at 11:38
  • This question and discussion is closely related to this other question on SO. – Paulo Merson Mar 24 '17 at 12:13
up vote 35 down vote accepted

The usual approach is to isolate those microservices as much as possible - treat them as single units. Then transactions can be developed in context of the service as a whole (ie not part of usual DB transactions, though you can still have DB transactions internal to the service).

Think how transactions occur and what kind make sense for your services then, you can implement a rollback mechanism that un-does the original operation, or a 2-phase commit system that reserves the original operation until told to commit for real. Of course both these systems mean you're implementing your own, but then you're already implementing your microservices.

Financial services do this kind of thing all the time - if I want to move money from my bank to your bank, there is no single transaction like you'd have in a DB. You don't know what systems either bank is running, so must effectively treat each like your microservices. In this case, my bank would move my money from my account to a holding account and then tell your bank they have some money, if that send fails, my bank will refund my account with the money they tried to send.

  • 3
    You are assuming that refund can never fail. – Sanjeev Kumar Dangi Nov 1 '17 at 11:11
  • 5
    @SanjeevKumarDangi it's less likely and even if it does fail, it can be easily handled since the the holding account and the actual account is in the bank's control. – gldraphael Dec 27 '17 at 10:29

I think the standard wisdom is to never have transactions cross microservice boundaries. If any given set of data really needs to be atomically consistent with another, those two things belong together.

This is one of the reasons it is very hard to split a system into services until you have fully designed it. Which in the modern world probably means written it...

  • 25
    Such approach could very well lead to merging all microservices to a single monolithic application in the end. – Slava Fomin II Mar 23 '17 at 14:55

I think that if consistency is a strong requirement in your application you should ask yourself if microservices is the better approach. Like Martin Fowler says:

Microservices introduce eventual consistency issues because of their laudable insistence on decentralized data management. With a monolith, you can update a bunch of things together in a single transaction. Microservices require multiple resources to update, and distributed transactions are frowned upon (for good reason). So now, developers need to be aware of consistency issues, and figure out how to detect when things are out of sync before doing anything the code will regret.

But maybe in your case, you can sacrifice Consistency in pos of Availability

Business processes are often more tolerant of inconsistencies than you think because businesses often prize availability more.

However I also ask myself if there exists a strategy for distributed transactions in microservices, but maybe the cost is too high. I wanted to give you my two cents with the always excellent article of Martin Fowler and the CAP theorem.

  • 1
    In distributed business transactions, consistency is sometimes addressed by adding an orchestration layer like a workflow engine or a carefully designed queue. That layer handles all the two phase committing and rolling back, and lets the microservices focus on specific business logic. But back to CAP, investing in availability and consistency makes performance the victim. How do microservices compare to lots of decoupled classes comprising your business in OOP? – Greg Jul 27 '15 at 14:02
  • You could use RAFT via microservices to address consistency FWIW – f0ster Jan 8 '17 at 2:34
  • +1. I often wonder why in a microservices context transactions are desired at all, if all 'pull' views of the data can be implemented as materialized views. For example, if one microservice implements debits from one account and another credits to another account, the views of balances would only consider pairs of credits and debits, where unmatched credits and debits would still be in the buffers to the materialized view. – Sentinel May 23 at 13:09

As suggested in at least one of the answers here but also elsewhere on the web, it is possible to design one microservice which persists entities together within a normal transaction if you need consistency between the two entities.

But at the same time, you might well have the situation where the entities really do not belong in the same microservice, for example, sales records and order records (when you order something to fulfill the sale). In such cases, you may well require a way to ensure consistency between the two microservices.

Traditionally distributed transactions have been used and in my experience, they work well until they scale to a size where locking becomes an issue. You can relax the locking so that really only the relevant resources (e.g. the item being sold) are "locked" using a state change, but this is where it starts to get tricky because you are entering the territory where you need to build all the logic to do this, yourself, rather than having, say a database handle it for you.

I've worked with companies that have gone down the route of building their own transaction framework for handling this complex issue, but I don't recommend it because it is expensive and takes time to mature.

There are products out there which can be bolted on to your system which take care of consistency. A business process engine is a good example and they typically handle consistency eventually and by using compensation. Other products work in a similar way. You typically end up with a layer of software near the client(s), which deals with consistency and transactions and calls (micro)services to do the actual business processing. One such product is a generic JCA connector which can be used with Java EE solutions (for transparency: I'm the author). See http://blog.maxant.co.uk/pebble/2015/08/04/1438716480000.html for more details and a deeper discussion of the issues raised here.

Another way of handling transactions and consistency is to wrap a call to a microservice into a call to something transactional like a message queue. Take the sales record/order record example from above - you could simply let the sales microservice send a message to the order system, which is committed in the same transaction which writes the sale to the database. The result is an asynchronous solution which scales very well. Using technologies like web sockets you can even get around the problem of blocking which is often related to scaling up asynchronous solutions. For more ideas on patterns like this, see another of my articles: http://blog.maxant.co.uk/pebble/2015/08/11/1439322480000.html.

Whichever solution you end up choosing it is important to recognize that only a small part of your system will be writing things that need to be consistent - most access is likely to be read-only. For that reason, build the transaction management only into the relevant parts of the system, so that it can still scale well.

In microservices there are three ways to achieve consistency between diff. services:

  1. Orchestration -- One process which manages the transaction and rollback across services.

  2. Choreography -- Service pass messages between each other and finally reach a consistent state.

  3. Hybrid -- Mixing the above two.

For complete reading go to the link : https://medium.com/capital-one-developers/microservices-when-to-react-vs-orchestrate-c6b18308a14c

I would start with decomposing problem space -- identifying your service boundaries. When done properly, you'd never need to have transactions across services.

Different services has their own data, behavior, motivational forces, government, business rules, etc. Good start is to list what high-level capabilities your enterprise has. For example, marketing, sales, accounting, support. Another starting point is organizational structure, but be aware that there is a caveat -- for some reasons (political, for example) it might be not the optimal business decomposition scheme. More strict approach is Value chain analysis. Remember, your services can include people as well, it's not strictly software. The services should communicate with each other via events.

The next step is to carve these services. As a result you get still relatively independent aggregates. They represent a unit of consistency. In other words, their internals should be consistent and ACID. Aggregates communicate with each other via events either.

If you think that your domain demands consistency first, think again. None of the big and mission-critical systems are built with this in mind. They all are distributed and eventually consistent. Check Pat Helland's classic paper.

Here are some practical tips about how to build a distributed system.

There are many solutions that compromise more than I'm comfortable with. Granted, if your use case is complex, such as moving money between different banks, more pleasant alternatives may be impossible. But let's look at what we can do in the common scenario, where the use of microservices interferes with our would-be database transactions.

Option 1: Avoid the need for transactions if it all possible

Obvious and mentioned before, but ideal if we can manage it. Did the components actually belong in the same microservice? Or can we redesign the system(s) such that the transaction becomes unnecessary? Perhaps accepting non-transactionality is the most affordable sacrifice.

Option 2: Use a queue

If there is enough certainty that the other service will succeed at whatever we want it to do, we can call it via some form of queue. The queued item won't be picked up until later, but we can make sure that the item is queued.

For example, say that we want to insert an entity and send an e-mail, as a single transaction. Instead of calling the mail server, we queue the e-mail in a table.

Begin transaction
Insert entity
Insert e-mail
Commit transaction

A clear drawback is that multiple microservices will need access to the same table.

Option 3: Do the external work last, just before completing the transaction

This approach rests on the assumption that committing the transaction is very unlikely to fail.

Begin transaction
Insert entity
Insert another entity
Make external call
Commit transaction

If the queries fail, the external call has not taken place yet. If the external call fails, the transaction is never committed.

This approach comes with the limitations that we can only make one external call, and it must be done last (i.e. we cannot use its result in our queries).

Option 4: Create things in a pending state

As posted here, we can have multiple microservices create different components, each in a pending state, non-transactionally.

Any validation is performed, but nothing is created in a definitive state. After everything has been successfully created, each component is activated. Usually, this operation is so simple and the odds of something going wrong are so small, that we may even prefer to do the activation non-transactionally.

The greatest drawback is probably that we have to account for the existence of pending items. Any select query needs to consider whether to include pending data. Most should ignore it. And updates are another story altogether.

Option 5: Let the microservice share its query

None of the other options do it for you? Then let's get unorthodox.

Depending on the company, this one may be unacceptable. I'm aware. This is unorthodox. If it's not acceptable, go another route. But if this fits your situation, it solves the problem simply and powerfully. It might just be the most acceptable compromise.

There is a way to turn queries from multiple microservices into a simple, single database transaction.

Return the query, rather than executing it.

Begin transaction
Execute our own query
Make external call, receiving a query
Execute received query
Commit transaction

Network-wise, each microservice needs to be able to access each database. Keep this in mind, also in regard to future scaling.

If the databases involved in the transaction are on the same server, this will be a regular transaction. If they are on different servers, it will be a distributed transaction. The code is the same regardless.

We receive the query, including its connection type, its parameters, and its connection string. We can wrap it up in a neat executable Command class, keeping the flow readable: The microservice call results in a Command, which we execute, as part of our transaction.

The connection string is what the originating microservice gives us, so for all intents and purposes, the query is still considered to be executed by that microservice. We are just physically routing it through the client microservice. Does that make a difference? Well, it lets us put it in the same transaction with another query.

If the compromise is acceptable, this approach gives us the straightforward transactionality of a monolith application, in a microservice architecture.

protected by gnat May 30 '17 at 16:14

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.