Best practices for delayed actions in event-driven architecture

Question

I'm working on an event-driven system and I'm facing a problem I never considered before.

There is a particular user command - let's call it "A" - that must be processed possibly with a delay of 10 minutes because user wants to be able to cancel that action issuing a new command - say "C" - until that delay has passed. Obviously, if the user issues "C", the delayed event must not produce any effect. This mechanism resembles the unsend message feature of Gmail.

My problem is mainly how to cancel the unwanted event, because events are immutable and cannot be deleted. The following scenario, for example, would fail even if I would check the current status of all the entities involved:

1. User issues command "A" → created event "EA1" with 10 mins delayed effect
2. After 8 mins, user issues action "C" → created regular event "EC"
3. Immediately after, user issues action "A" again → created another event "EA2" with 10 mins delayed effect
4. After 2 mins, the event "EA1" will produce its effect, but 8 minutes early from the user's point of view, and I cannot avoid it because the status of the entities involved is the same of after point 1.

Is there a best practice to manage cancellable actions in an event-driven architecture?

Thanks.

Answer 1

My problem is how to cancel the unwanted event because events are immutable and cannot be deleted.

Events are immutable because they are past. If you are delaying or scheduling events, you are doing something wrong.

After 2 mins, the event "EA1" will produce its effect,

This doesn't seem right²; maybe you are conflating events with actions. Check out the following sequence of actions and events:

• Command A is a request to schedule^(action) task
  • The scheduler¹ casts TaskScheduled^(event) immediately after scheduling the task.
  • TaskScheduled handlers enable C commands for the next 10 minutes.
• Command C is a request to cancel^(action) a concrete scheduled task.
  • The scheduler casts TaskCancelled^(event) immediately after removing the task or invalidating the timer.

If task is not cancelled, it gets executed^(action) after 10 minutes and TaskCompleted^(event) is cast at the end. This last event, like any other, can't be modified.

In other words, commands A and C target the state of the scheduler, not the state of a non-existing event.

To "revert" or "nullify" an event, we can execute a compensatory action, which is the opposite action of the original one. In other words the command !A. The problem is, not all actions have a compensatory action. For example, we can't revert notifications.

---

^{1: Whatever is holding and managing the timers
2: Events aren't: request, commands, actions, tasks, messages,}

Answer 2

You handle every event. Sometimes you handle it by doing nothing. That’s the case here.

Event A triggers a ten minute timer and sets a state to “waiting for timer”. Event C sets A’s state to “do nothing” in state “waiting for timer” and does nothing otherwise. The timer starts the action and changes the state to “action started” in state “waiting for timer” and does nothing otherwise.

Once the action for A starts you should make C impossible in your UI. And if event A can happen multiple times then figure out what you want to happen.

Answer 3

Although you could probably solve this with something like Kafka Streams I would suggest keeping the solution simple:

• Create (or integrate) a scheduling service.
• Each time a new "delayed" event arrives, it is added to the datastore for the service.
• If a cancel event occurs before the "firing" time of the event, cancel the event.
• If no cancel is received (before the firing time), fire a new message (to a different queue) which will then be processed by the service that is actually going to do the work.

If you integrate something like Quartz it will deal with polling its datasource looking for when to fire events. If you built it yourself you will need a worker thread that checks periodically (say once per minute) to see if any new events need to be fired.

Edge Cases

Out of order Events

If the original event and the cancel are chronologically very close, it is possible the original and cancel events occur out of order. To mitigate you can store the cancels in case a later event occurs that should be matched (you will need to timestamp the events close to the user so that you know which occurred first).

Client Notifications

When the scheduling service receives a cancel event there are three possibilities:

• The original event is found and cancelled.
• The original event has fired and cannot be cancelled.
• No matching original event is found.

I would suggest you send a response back to the client in all three cases. It is obvious how to handle the first two cases (just notify the user of what happened), in the last case you probably want to let the user know that the cancel has been registered, however the disposition of the cancel isn't known yet.

If a later event arrives, matches (actually occurred before the cancel) and is cancelled you can then send an additional notification to the user that the event has now been cancelled.

Multiple Events/Cancels

Unless you need something more complex, I would suggest that you record the time of each event (original & cancel) when it first arrives. That way you can simply cancel any original event that was received before a given cancel (assuming it hasn't executed yet). Hence a single cancel may cancel multiple events.

Answer 4

Service A is sending a command CMD to service B. We also need to consider other services XYZ that may be concerned by CMD and need to receive the message.

There are 2 cases:

• would XYZ need to know about CMD when it's issued (before the 10 minutes delay),
• or when it's executed (after the 10 minutes delay)?

Case 1: the command does not need to be received by other services until it's actually executed

If there is no other service that needs to receive CMD before it's actually executed - then you should simplify and handle the delay internally within service A.

Service A would internally manage the command ID, handle cancelation, and when the delay is passed and no cancelation was received, it will broadcast the CMD message for immediate execution by service B.

Case 2: the command needs to be received by services XYZ at the time it's issued

If other services need to be aware of the command CMD when it's issued, then you need to have a scheduler outside of service A.

The scheduler could be within service B or could be a new service, which even makes more sense depending on your other requirement and your current architecture

Service A should assign a UUID to CMD, so service A can issue further messages related to this command (ex: cancel command with UUID).

There is an important element to consider: when the countdown should actually start? Would it be the time when CMD is issued or the time when CMD is received by the scheduler?

In the first case, if there is a delay (ex: the scheduler is down for 10 minutes), then the scheduler would receive CMD after the delay is elapsed and fire it for immediate execution without checking for any cancelation messages pending in the queue. So you'll need a mechanism against this.

In the second case, if there is a delay (same example) the scheduler would start the 10 minutes countdown after it's coming back up, so the CMD would be delayed by more time than expected. So you need to be sure it's acceptable.