0

As this article explains the way to deal with persistent state in an event log driven architecture is to have some "microservice" subscribed to events and be responsible for maintaining that state, such as tallies of the weight of all green trains in the last 7 days etc. But I'm uncertain about how to implement conditional logic with an event log where the validity of one event depends on other events happening first.

Scenario: I have a process in which first an object is registered and given an identity. Then at some later point the object can be processed. But the processing validation needs to know that the event describing the processing refers to a registered object. So I need to maintain a database of registered objects by "event-sourcing". But there is two alternative approaches to implementing the conditional validation for the processing step:

  1. Put the conditional check in a microservice that receives (possibly invalid) "processing" events, checks its event-sourced database of registered events then generates another event if it's registered or doesn't if it's not.
  2. Just have some API in front of the processing event source that checks a database of registered objects from a database source from the event log, and it will never generate an event in the first place if the object isn't registered.

Maybe it's splitting hairs but in #1 the processing event source and the validator both interface with the event log only. Where as in #2 we have a ~front controller that knows about the queue, a database of registered objects, and how to validate requests to create a new processing event. Which is better? Would experience EDA architects prefer on over the other?

2
  • 1
    Your 1 & 2 are literally the same thing. Nov 7, 2021 at 4:08
  • @FilipMilovanović Not quite. I added a paragraph to clarify. Nov 7, 2021 at 4:58

2 Answers 2

1

The goal (normal case) should be to have all events pertaining to an object (lifecycle as well as processing events) in one ordered event stream such that invalid processing events just don't happen. Every microservice dealing with processing events for objects should check their lifecycle events, too, so it can decide on its own whether an object exists and is in a valid state for the event being handled. Don't create additional dependencies to separate services (whether as APIs or as event stream filters) which only make your system complex and brittle by introducing additional possible points of failure.

So I would consider a third option: the processing event source should be a lifecycle event consumer, maintaining its own registry of valid objects. Then it doesn't depend on an external API, and the event processing service does not depend on an event filter. Both would be able to catch invalid events at the first possible opportunity, and avoid generating invalid events themselves.

1

I'm uncertain about how to implement conditional logic with an event log where the validity of one event depends on other events happening first.

This spelling is a little bit odd. "Events" (in the context of event sourcing) are messages that describe information that has changed in the past. So it really doesn't make sense to talk about events as valid or invalid relative to some other state you have lying around - the change of information described by the event has already happened.

Instead, we talk about the states of the subscribers of an event, which is normally determined by what information they have seen.

We can think of the subscriber as a Mealy state machine. If it is in the state where it has already seen a FooRegistered(id:12345), and it now sees `FooProcessingAuthorized(id:12345, authorizedBy:Bob), then it kicks itself into a new state and fires the processing action.

But in the state where FooProcessingAuthorized is published, but it hasn't (yet) seen FooRegistered... well, that's a different state transition, and there are a lot of choices you might make. You might immediately go into a Failed state, you might instead go into a pending state where the action is to schedule a timeout event (waiting to see if the FooRegistered event arrives "later").

In a distributed system, information isn't instantly available everywhere; if you want a robust design you have to take that into consideration.

Among other things, that means that while you can put in gates to ensure that Bob doesn't authorize processing when he can't yet see that a Foo is registered, that doesn't guarantee that the processor will be able to see the registration when the authorization arrives.

So you can get a lot of nines of reliability by adding a check, but you don't get infinite nines.

In some designs, a single ordered event stream is used for everything, which gives you certain guarantees (any subscriber who can see event(id:3) is guaranteed to be able to see event(id:1) and event(id:2) by incurring some additional costs (contention, availability).

1
  • "it really doesn't make sense to talk about events as valid or invalid relative to some other state you have lying around - the change of information described by the event has already happened.". Yeah this is the conventional thinking that makes sense to me: You have some state machine / data model you poke it to change its state this generates events right? So those events have to consistent. But I thought event driven architecture was different where you don't actually have a state machine / data model, you just have "first and foremost" an immutable event log? Nov 15, 2021 at 2:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.