How would an event driven architecture deal with agregating events over a time period?

Question

Say I have a service that is producing events of the format

type: SEND_MESSAGE, 
to: '[email protected]', 

And I have another service, that is saying, "for the next four hours, I want to consume all SEND_MESSAGE events that are directed to the people in this array: ["[email protected]", "[email protected]", [email protected]"]".

So ok, this consumer can just consume the queue and ignore any messages that it doesn't care about.

However, at the end of the four hours, it also wants to know 'Which people in this list didn't I get messages for, I want to do some kind of action based on that'.

How would this be achieved with an event-driven architecture?

As far as I know, there are three kinds of approaches:

1. You have some kind of database/persistent store that the second service uses to keep track of which people it has received messages for.
2. Either the consuming service, or another service, that creates a 'people who have been sent messages' aggregation event.
3. You have an event queue that supports 'look back in time', and you just examine the last four hours of events.

On this third point - my understanding is that some event queues/streams support this, and others being necessarily ephemeral - (ie. services have to deal with the event in the moment, or miss it forever). Or is this a misunderstanding on my part?

What I'm wondering, is what are the key concepts and standard patterns used in event driven architecture I would use to solve this problem?

Answer 1

I'd opt for number 1. This follows more or less the CQRS pattern and you have a Query here ("which people in this list didn't I get messages for") and as such this should be handled separately. The repository that handles this could be simply in-memory or in a dedicated db. Building, mocking and testing it can be ease. The amount of data you need to aggregate is very little, just a list of names.

Option 2 clutters your domain with hyperspecific query messages, which doesn't make sense.

Option 3 is just a variation of 1 in my opinion. You need to store the data for 4 hours and then you run a query on it. Depending on how many messages this can be very inefficient. Again, the amount of data you need to aggregate is very little, just a list of names, so why would you buffer everything?

Answer 2

The use case of querying events over time windows, looking for patterns of events, in combination or in aggregate is the realm of Event Streaming Processing. Take a look at technologies such as Esper or Kafka Streams.

Esper for instance, allows you write queries which select events in combination from a stream over a time period and take appropriate actions. A typical use case would be fraud detection which might look for patterns of events over rolling time periods to spot unusual activity.

It looks like you want to process events one at at time, on the one hand, but also be able to look back and query the stream to see what events have arrived over a time period. Event Streaming Processing would be my first port of call as you could support your use cases with the event stream alone without the need of additional layers.