I am reading “Building Event Driven Microservices”.

It states a requirement of the event broker is:

Infinite retention: Event streams must be able to retain events for an infinite period of time. This property is foundational for maintaining state in an event stream.

However this does not seem to be standardly available. For example, Azure Event Hubs has a maximum retention policy of 7 days, and Event Grid also has maximum retry and retention time policies.

What are techniques for coping with retention limitations of the event broker technology that is used?


When building event driven systems, events might be used for two distinct aspects of the system:

  • for communication between components, e.g. using a message queue
  • for representing the state of a service

These concerns are related, sometimes intertwined, but often technically distinct.

Infinite retention of an event stream would typically mean that you store the events in a database, allowing you to replay them and to rebuild the current state at any later point in time. This property of an event-based system has nothing to do with how your services communicate – it works just as well regardless of whether you use message queues or REST as a communication pattern.

Certain message queue systems do have the ability to store messages/events indefinitely. Notably, Kafka has this capability (and cloud providers will happily sell you this extended retention). But it's not necessarily wise to use such capabilities.

Communication in a distributed system must be assumed to be unreliable. If your system only works correctly if messages are delivered exactly-once, you're going to have problems.

E.g. assume that you have a horizontally-scalable service where every instance has its own cursor for consuming from the message queue. If one instance is temporarily turned off for half a year and then restarts, it will likely spend multiple days just catching up with old messages. The ability to rebuild the entire state can be very important, but might be a poor practice when used as a standard operating procedure – even in an event-driven system, you likely want checkpoints of the current state that can be shared between instances.

Dealing with limited-retention message queues can be an “enabling constraint”: forcing you to think about what happens when messages are dropped, and about how state can be rebuilt efficiently.

In a simple message-oriented service architecture, we might have two services A and B that are connected via some queue Q. When A sends a message to B, what does this mean, what can A assume about B? A should assume absolutely nothing. A should not assume that B will receive and process the message within any particular deadline. If the message is important, A might try re-sending the message with exponential backoff until B sends another message in response (or use a message queue that guarantees delivery attempts until B acknowledges it). If B wants to be able to re-build its own state in an event-driven manner, it might store events in its own database – but would probably not rely on the message queue Q for persistent event storage.

In practice, it is possible to make message queues (and any distributed system, really) highly reliable so that guaranteed delivery can be ensured almost always. But whether that's the case depends on the Ops team that maintains this architecture, and on the specific guarantees provided by your specific technology choices.

  • I guess it is either A’s responsibility to retry until receiving an ack message from B, or B’s responsibility to periodically request all events / snapshot from A to check for missed events? The latter would be more suitable for multi-consumer.
    – MattHH
    Sep 5 at 3:26
  • 1
    @MattHH Either might be reasonable, depending on context. Ultimately, some component will have to handle retries, you just have choices about which component (e.g. A, B, or Q).
    – amon
    Sep 5 at 9:39

To simply fix the issue at hand (data retention needs to be increased) one alternative is switching to service that allows for infinite retention (say Apache Kafka), however be aware that one day might run out of resources / maintaining this amount of resources would be too costly.

The other is analogous ot tiered storage - the last X days are easily accessible through your current stack, the older data is acccesible through some other higher-capacity/cheaper technology.

Of course the latter scenario brings in multiple pain-points:

  • the user needs to communicate with more services (current and new backend) OR you need to setup an aggregator service in the middle to do the same,
  • migration of data from the first to the second storage - how reliable should it be, can the consumer tolerate duplicates if the migration is not transactional.

Infinite retention might be the ideal goal, but it is unrealistic. It's like to say that everything can be done with infinite money and infinite resources.

That said if you have a service that requires a long retention time you setup by yourself the messaging system according to your need, any system base on queues would suit. But before you have to do a proper estimate of the amount of space and memory you might need and you have to take into account that eventually you will reach a limit anyway. So, how to cope with the limit? A lot of queue based system allow to define an action to be executed when an event expires (see TTL). It can be managed as an error.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.