2

I have microservice architecture. I implement a distributed transaction flow (2-phase commit), kafka, and 1 coordinator service. So, my problem is in each step of preparation I am calling an outside API. If that API succeeds, everything is ok and preparation will happen. But if something happens during those preparation steps, I cannot redo external API changes. For example, each step may call bank APIs. So if one of these steps fail I don't have a redo mechanism for those changes. What should I do?

8
  • 1
    I changed the abbreviation "2pc" to expand it out to "2-phase commit". If that is not what you mean, feel free to roll back my edit. It wasn't an abbreviation I was familiar with and I had to look it up. I figured others might have the same question. Commented Jun 21 at 20:34
  • @GregBurghardt thank you for editing. I think my answer could be related to the Circuit breaker pattern. But I am unsure if it is enough or if I am missing something. Commented Jun 21 at 21:01
  • 4
    That external API is not part of transaction. In other words you don't have, and likely won't ever have 2pc. If you cannot redo or undo those changes later, then you don't even have eventual consistency. Its like saying: "I can only write to a file, but what happens if I need a rollback?". I'm sorry, you are f***ed.
    – freakish
    Commented Jun 21 at 21:04
  • "So if one of these steps fail I don't have a redo mechanism for those changes." — do you mean undo or really mean redo?
    – Erik Eidt
    Commented Jun 21 at 23:39
  • Btw, 2pc would have each service report if they can succeed, yet hold off final commitment until all succeed or one fails (and that would have to accomplish undo in the case of failure).
    – Erik Eidt
    Commented Jun 21 at 23:41

1 Answer 1

9

I don't believe you will find a purely technical solution to this. You will need to keep track of which calls to external APIs have succeeded, and then build logic to repeat the workflow, but skip the steps that have already succeeded. There will inevitably be cases where you cannot write code to fix this, in which case you will need to record what you did, what went wrong, and then provide screens in your application for human beings to attempt to undo things manually. For example, if you charge someone's credit card for a service, the customer service personnel will need a means to refund that charge.

Most likely you will need a combination of defensive programming, retry mechanisms and changes to your business flow to accommodate all of your failure modes. This won't just be a technical fix. It should involve business workflows as well, so you will need to work closely with product management, end users, and stakeholders to identify a robust solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.