I have a Kafka topic providing events of the following type:
- These events may be generated at a high rate and are idempotent, i.e. getting
123:ADDone time and ten times in a row produces the same result: state of
- Processing of each event may take a while (not hours, but seconds).
- Processing of each event may generate new events for different keys. Ex:
123:ADDmay produce secondary event
234:REMOVEor any other number of
- For same keys secondary event should be overridden by primary event if its parent was received before the overriding event.
- Model state should represent result of the last event received at the moment.
Amount of events is significant, so in a linear single-threaded way this is too slow. Multiple processor instances are expected.
While I can use Kafka partitioning by key to assure primary events are processed in the right order, I'm not sure how to handle the appearing secondary events correctly.
|secondary 234:REMOVE is produced while processing the first event.
In this case final state of
234 would be
REMOVED however as the parent of secondary event was received before primary
234:ADD the order of event processing should be different and the final state should be
I thought about adding timestamps to events and making secondary events get same timestamp as their parents. This way it would be possible to discard outdated events.
My questions are: how do I sync all the event processors across multiple instances of a service to guarantee that event with an earlier timestamp is not executed after event with a later timestamp?
How do I guarantee that timestamps are in sync across instances of service?
I might push all the events to some db, but making sure that no event with given key exists at the moment would require a full table lock which to me looks as no good for concurrent access.
So I'd be grateful for any hints or ideas or proving me wrong.
Solution I'm thinking of at the moment:
- All the primary events I'm processing are ordered in kafka, so it would be fair to produce these events with a sequentially incrementing timestamp or id. Second option is to use Kafka offset to order the events.
- I can add parent's timestamp/offset to spawned events as I know exactly who spawned it.
- After processing these events I can propagate them to a different kafka topic partitioning by key.
- I can add another service consuming data from second topic and adding pairs
key/timestampto a database.
- If the event I'm consuming has an earlier timestamp, drop it, otherwise update timestamp in db and perform action.
Partitioning in Kafka guarantees all events with same key are handled by the same consumer and this way may be processed in linear mode. At the same time events with different keys may be processed in parallel. Clock sync shouldn't be an issue in this case as all the timestamps/offsets are set by an ordered primary producer.
This way there even may be no need in table locks as concurrent access to same data shouldn't happen by design.