0

We have a bunch of services which are using AWS Kinesis to communicate via events. We have different events such as MasterDataUpdatedEvent, PriceUpdatedEvent or AvailabilityChangedEvent.

Currently, we are using a dedicated stream per event:

  • MasterDataUpdatedEvents go to master-data-updated-event-stream
  • PriceUpdatedEvents go to price-updated-event-stream
  • AvailabilityChangedEvents go to availability-changed-event-stream

So we know exactly what's on each stream. However, configuration is a huge burden and there will be much more events in the end.

Are there any best practices for this case? Would it be "OK" to send everything to a single stream?

2

It depends on how your consumers process the streams.

Every consumer processes each type of event, independently

This might be a typical logging stream: you care that the events happen, but aren't performing any processing that needs to look at multiple events.

In this case combining the streams makes sense: you have a single reader per consumer, and save offsets after each event or group of events.

A consumer consumes only a single event type, and needs to perform some processing over groups of events

This might be the case where you're using events to populate downstream databases. For efficiency you'll want to gather multiple events for a single update, and you may need to reduce the events in some way (for example, you may have multiple events for a product but only care about the latest).

In this case separate streams make more sense. You could write your consumers to simply discard events that they don't care about. However, you have a fixed bandwidth per reads: 5 transactions or 2MB per second per shard. It's really easy to use up this bandwidth just with consumers that care about all of the records; discarding records could make your system unacceptably laggy (or could push you toward fan-out implementations, which you'd find even more complex to manage).

A consumer consumes multiple event types, in batches

This case is similar to the previous, but assumes some sort of inter-relatedness between the event types. For example, you want to combine different types of events in a single transaction.

In this case it might make sense to combine the streams, and rely on related data being written together. In practice, it's way too easy for the shards to get out of sync, meaning that you're buffering data and you have long waits to save offsets. In this case I think Kinesis is probably the wrong solution.

0

Here is how you achieve this...

  1. Decorate the event json with a wrapper that identifies the event type
  2. Have a default stub lambda function process all messages
  3. Direct to different function based on the event

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.