1

I’m building a microservice which depends on a set of downstream services to collect the data necessary to operate. It is a data delivery system, I’m not source of truth for any data, my system rather aggregates data and serves it. The downstream services are sending their data through kafka streams. Nevertheless some of them are already in place for quite some time and they dont guarantee that their current internal projections are totally in sync with the kafka beginning of history. The consumer group of kafka also just guarantees some weeks of retention. Over that period you cannot depend on kafka to load the older version of your messages. One would need to go to s3 buckets to fetch them.

I would appreciate suggestions and good practices in situations like this to the following scenarios:

  • How to build the first projection of the data?
  • If I lose all of my data, how should I rebuild the current state?

As a integrator of this downstream services I see a clear option:

My Option: Import their first initial state manually and build recoverable snapshots of my database, which have a latest imported offset. Once I have to disaster recover I load only the most recent messages based on that offset.

Nevertheless I’m not sure if I’m missing options or if there are good practices in this situations, in general and in specific to kafka, or if I’m not doing an anti-pattern. Suggestions?

1
  • 2
    Event sourcing depends on the fact that you start from the beginning and build your model from there (unless you have some snapshots along the way, which are guaranteed to be in a correct state). If you need to model a new projection, I don't think there is another option besides going to the beginning of the event stream and replaying it. It's not uncommon that spinning up a new projection can take hours, if not days, to complete.
    – Andy
    Jul 10, 2020 at 11:57

1 Answer 1

1

You can have an event at the start of the stream that is an Onboarded event with the initial state as it became known to your event sourced system.

You would need all your projectors to handle this event appropriately to use it as the starting point of their state projection.. for example if a bank account is onboarded with a beginning balance then all the withdrawals and deposits need to be applied on top of that initial state in order to get the current state.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.