1

Say I have a service that is producing events of the format

type: SEND_MESSAGE, 
to: '[email protected]', 

And I have another service, that is saying, "for the next four hours, I want to consume all SEND_MESSAGE events that are directed to the people in this array: ["[email protected]", "[email protected]", [email protected]"]".

So ok, this consumer can just consume the queue and ignore any messages that it doesn't care about.

However, at the end of the four hours, it also wants to know 'Which people in this list didn't I get messages for, I want to do some kind of action based on that'.

How would this be achieved with an event-driven architecture?

As far as I know, there are three kinds of approaches:

  1. You have some kind of database/persistent store that the second service uses to keep track of which people it has received messages for.
  2. Either the consuming service, or another service, that creates a 'people who have been sent messages' aggregation event.
  3. You have an event queue that supports 'look back in time', and you just examine the last four hours of events.

On this third point - my understanding is that some event queues/streams support this, and others being necessarily ephemeral - (ie. services have to deal with the event in the moment, or miss it forever). Or is this a misunderstanding on my part?

What I'm wondering, is what are the key concepts and standard patterns used in event driven architecture I would use to solve this problem?

2
  • Do your case suppose that your service that will work for 4 hours may go down or may need to scale up by getting multiple instances working on it ? Or is your service alone and we suppose it won't fail ? If we suppose a full resilient architecture, you will need in fact to have consumers that transform this event into another one, then the service can consume them and emit events of what he has done. Then when an event indicating the 4 hours of working are done, an event is emitted and a service will read what has been done in those 4 hours to perform appropriate action.
    – Walfrat
    Commented Jun 26, 2020 at 8:59
  • Surely a simpler example can be used, such as counting the number of messages. Commented Mar 23, 2021 at 18:06

2 Answers 2

1

I'd opt for number 1. This follows more or less the CQRS pattern and you have a Query here ("which people in this list didn't I get messages for") and as such this should be handled separately. The repository that handles this could be simply in-memory or in a dedicated db. Building, mocking and testing it can be ease. The amount of data you need to aggregate is very little, just a list of names.

Option 2 clutters your domain with hyperspecific query messages, which doesn't make sense.

Option 3 is just a variation of 1 in my opinion. You need to store the data for 4 hours and then you run a query on it. Depending on how many messages this can be very inefficient. Again, the amount of data you need to aggregate is very little, just a list of names, so why would you buffer everything?

0

The use case of querying events over time windows, looking for patterns of events, in combination or in aggregate is the realm of Event Streaming Processing. Take a look at technologies such as Esper or Kafka Streams.

Esper for instance, allows you write queries which select events in combination from a stream over a time period and take appropriate actions. A typical use case would be fraud detection which might look for patterns of events over rolling time periods to spot unusual activity.

It looks like you want to process events one at at time, on the one hand, but also be able to look back and query the stream to see what events have arrived over a time period. Event Streaming Processing would be my first port of call as you could support your use cases with the event stream alone without the need of additional layers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.