0

My application contains webhook handlers that should execute many actions when the event happens. Given CUSTOMER_REGISTERED event this is what happens in pseudocode:

function WebhookHandler(EVENT) {

const selectedStrategy = selectStrategyBasedOn(EVENT)
const result = executeStrategy(strategy);

...
}

function executeStrategy(event) {
  if (event === "CUSTOMER_REGISTERED") {
    const { token, domain } = getSessionDetails(event)
    if (!token && !domain) {
      printError()
    }

   const systemAResult = getMemberFromSystemA(token, domain)
   
   if (!systemAResult) {
     ...
   }

   const systemBResult = saveDataToSystemB(event.member)

   if (!systemBResult) ...

   const systemCResult = getAllMembersFromC()

   const saveResult = saveMemberInASystem(systemCResult.members.filter())
   
  }
}

This is just an example that is supposed to show you that a lot can happen when the event is passed to executeStrategy(). Some of these can be optimized and put into an async call but some depend on each other. Some functions are simple getters but there are setters. The problem with this approach is:

  1. This is somewhat of a snake code. I'm saving the result in the constant, and then in the next line, I'm passing it as the parameter. So it goes from left to right. Each time I have to come up with a variable name just for that purpose.
  2. Error handling is just ugly
  3. If the whole process is not complete then it's unsuccessful. But some data were already saved in the first system but not in the second one. I need to reset all calls to the previous state.

The third point creates new challenges:

  • what if undo function fails? Should I retry?
  • how do I determine which undo function should I call?

I am looking into the chain of responsibility and pipeline pattern but I'm not convinced if it's going to work in that case. I am writing code in JS.

Thanks

2 Answers 2

3

Seems to be a classical "transactional" problem.

When a event is handled, a bunch of activities should be executed, but only in total, not partially if some parts fail.

So, first i would try to run as much of that code that has no sideeffects as possible at the beginning. That means all that "get data, calculate stuff, ..." code. And keep the necessary data in memory. I translate that for myself into "only changing my local state".

When that was successful, you will have to run the ugly part. Changing the state of external systems. Thats ugly because if somethings fails, the external state is already changed and you have to roll the change back. The roll back strategy depends on the implementation of those external systems.

Some allow transactions, that means you can "open" a transaction, then submit the changes, do other stuff and at the end "close" the transaction, when everything was successful. And only with the "closing" the other system really changes its state.

Other systems do not support transactions. That means you can only revert a change by adding another change. Therefore you have to think about your own rollback strategy, like "i created that entity, therefore as a rollback i have to delete it again". But thats quite error prone, because you may not know all the side effects, your initial change had. Like "adding that entity increases that special counter. but deleting it does not decrease it".

And some changes do not have an appropriate additional change to revert the initial one. For example if the external system itself updates other external systems because of your initial change.

Therefore as a rule of thumb, try to first do all that stuff which does not change anything (reading, calculating,...).
Then do that stuff that support transactions, so that you easily can cast away your changes, if something fails later.
Then do the stuff which needs "manual" reverting. And last but not least, at the end do the stuff which are irreversible.
That way you at least reduce the chance of issues.

If an changing action (which can fail) depends on information that you only get by another changing action (which is irreversible), than thats quite unfortunate. This could then only be solved by architectural changes, liike that the changing action does not depend anymore on information of the irreversible change action, or that the irreversible change actions changes to be not irreversible anymore... :-(

Now about your questions

"should i retry my undo function"
=> The issue is, that with remote functions, you often do not know if they failed AFTER or BEFORE they done the important stuff. Therefore idempotency (the ability to apply the exact same request mulitple times without altering the result) gets important. Think about you want to revert a change which increased a counter. Just decreasing it (run a -1) does not do the trick, because executing that request two times, may decrease the counter by 2. An alternative would be to take the counter value (it was X after the increasing) and now send a request to set it to X-1. That request could be send as often as we want, the result will always be the same. The downside is, that if OTHER requests manipulate the counter (and it was already even more increased) this idempotent requests really creates havoc. :-(
So the answer (as always) it depends.

"how do i determine which undo functions to call"
=> You have to keep track of which changing functions you have called in a transaction. And then, after one failed, you have to undo all those changes. This may, or may not include the failing change itself (depends on the error). When you run a function, it should already "prepare" the undo function (by storing the relevant parameters), then its easier to later run the undo functions.
But again, it depends on the functions. It may be a simple undo call, it may be a complex set of undo functions that have to be executed to revert a single simple change...
And be aware that the undo functions also can fail and therefore you may need a cleanup strategy for the cleanup.

Therefore its a good practice to think about "what could fail", "how likely is it to fail" and "how costly is it, if it fails".
And depending on that decide which solution is appropriate.

2
  • Your answer contains a lot of good recommendations. Unfortuenately it is actually hard to get through to since the formatting (or the absence of it) is so horrible, I give it a +1 either.
    – Doc Brown
    Sep 23, 2022 at 7:41
  • Wonderful, thank you!
    – filemono
    Sep 23, 2022 at 8:44
1

Just a minor thing: it is very helpful for the stability both of the system you are implementing, and of your mind, that an operation is atomic. That is, it is either executed successfully, or it returns an error, and has no side effects except perhaps logging things around the error.

So if you get partial results halfway through you should either not store them in a persistent place, or undo the store on a final failure.

3
  • Thanks. I think what you have meant by saying atomic is the amount of things that happen when the function is called. It should weather to its job and return the result or cause an error to be thrown and nothing else, right? The problem with store aspect is that systemA, systemB, systemC... and so on can act like a store. And things cannot be saved into systemB if they are not in systemA. Let's say systemA on succesfull save returns ID that is needed for systemB. And this is why we need undo step just in case
    – filemono
    Sep 22, 2022 at 22:00
  • 1
    I think in that case you should probably spend some time thinking about this snd discussing with colleagues whether and what problems it will cause if partial results are stored.
    – gnasher729
    Sep 25, 2022 at 14:20
  • Thank you @gnasher729
    – filemono
    Sep 25, 2022 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.