12

Is there a design pattern or practice I can use to help with services that are either down or go down, while others are stable?

What if I have three microservices, and two of them are good, and one dies right in the middle of a POST? Two will get the POST and one will not. I don't think I can do transactions because I'm shipping off my requests to a service.

How do I design for that? I don't want orphan data in various databases.

Improve this question
5
  • 6
    It's not a simple problem to solve. I have seen it implemented as queue's to the services (eventual consistency), since most likely, your not in control of the service(s), and imposing transaction managers or transactional capabilities is a crap shoot at best, and probably not a good idea in an SOA environment. I've mostly seen this around mobile push, where you may or may not have a connection to your destination.
    – Mike
    Dec 1, 2016 at 22:04
  • acid over microservices is a tough nut to crack, another option might be a bus of sorts, using redis publish/subscribe or a queue design and post once from the inbound channel, then your subscribing services or service proxies push to the targets and report success failure. You'll need to monitor for failures and have a flow for it too. You can also have failures where the transaction isn't valid on one service but valid on two others but its just another failure flow you'll need to address.
    – Tim Cederquist
    Dec 8, 2016 at 18:17
  • Wouldn't using something like "queue manager," which is what I guess Redis would be cause a bottleneck? Or at least have high potential too? I know of no other way than you described either.
    – johnny
    Dec 9, 2016 at 15:32
  • Depending on the volume of data flow, I have implemented a queue manager, that retries transmissions until success is reported or it posts a failed notification and sends an SMS alert about the outage. I guess it would depend a bit on the expected outage window as well (how long).
    – htm11h
    Jan 24, 2017 at 17:29
  • Is this what something like rabbitmq is for?
    – johnny
    Jan 24, 2017 at 18:35

2 Answers 2

Reset to default
9

Some options.

Use a persistent communication channel

Instead of HTTP, drop messages in a queue that is highly available and persistent. E.g. Kafka. As long as the target server becomes available at some point, it will get the message.

You have the trade-off of now provisioning and administering a complex subsystem (the queue). So make sure you analyze whether this is worthwhile.

Backoff and retry

Have the caller keep the failed request (possibly persisted to disk) and periodically retry. It's important in this case to distinguish between your request causing a crash vs the service just being down. The former is probably due to a bug and should be logged... retries probably won't make a difference until a fix is made.

Detect and compensate

A periodic task checks for consistency conditions between microservices. E.g. failure logs all the way up to direct API queries as necessary. If it discovers an issue (e.g. there's an order but shipping never received packing list) then do compensation steps. Those steps could be creating a support ticket for a manual fix, or emailing someone, or whatever.

Consider design alternatives

A case like this probably calls for an API gateway to manage calls to affected microservices. That way you control which tactics are used to mitigate this problem. You probably don't want to burden clients with those implementation details. See Circuit-breaker pattern.

Because microservices are independent, there will always exists some failure case that can result in inconsistency. You have to be prepared to do manual fixes when those arise.

If you require strong consistency, then microservices will not be a good fit. If still needing scalability, you might want to look into sharding where related data can be co-located on the same shard for consistency guarantees. You can still scale out IO by adding shards.

If you need strong consistency and aren't having scalability problems, then just use monolithic services. Use libraries as boundaries within your application to separate concerns.

Improve this answer
3
  • Is this what RabbitMQ is for?
    – johnny
    May 9, 2017 at 16:18
  • Is RabbitMQ the answer to your question? No. It could be a part of a solution that meets your needs, but it isn't going to solve your problem alone.
    – Kasey Speakman
    May 9, 2017 at 16:28
  • Just a note. I think RabbitMQ doesn't persist the messages. It's consumed and removed from the queue, so NO. If you need persistence and retry, RabbitMQ won't help.
    – Laiv
    May 9, 2017 at 20:45
2

I think what you're describing is the consensus problem: you don't want to commit unless each participant in the distributed transaction says the operation was successful. The simple solution to this is the Two Phase Commit. Essentially it stages the transaction in each system until each reports back that the staging was successful (Phase 1). If every participant in the transaction returns success, each are told to commit; if any of them instead returned a failure, a rollback is issued (Phase 2). There's a wrinkle to this that leads you to the more complex Three Phase Commit solution. You can read a much better description of each here:

http://the-paper-trail.org/blog/consensus-protocols-two-phase-commit/

http://the-paper-trail.org/blog/consensus-protocols-three-phase-commit/

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.