I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform.

I was wondering if it is common and/or considered a good practice to use the API Management as a gateway to the Kafka topics. The benefits could be analytics, caching, rate limiters and improved access control in the gateway layer (possibly others).

However, the communication between microservices is always a main topic in discussions. My main doubts are:

  1. When communicating, should my microservices call the internal APIs directly (via API Management) or should they produce/consume Events directly from Kafka topics? Is it an anti-pattern to have some microservices to expose APIs, but then some changes to the underlying data occur without making use of this API (i.e. directly in Kafka)?
  2. Since I'm using Event Sourcing and all microservices have access to this single source of truth, do I even need that much communication between microservices? I could duplicate the data that I need in local sources and only worry about communicating for transactions where several other microservices need to take action (e.g. a payment pipeline). Is this a good practice?
  • Not an answer to your question, but there is a concern with using Kafka for Event Sourcing. ES usually entails that your services need the ability to reconstitute the state of a single aggregate by reading the events for that aggregate. The most feasible way of doing that efficiently on Kafka would be to create a separate topic for each aggregate. However, topics are heavyweight objects - you don't want to have millions of them. Event Sourcing on Kafka works if you have relatively few (1000's), long-lived aggregates, but it is problematic if you have large amounts of aggregates. – Allard Dec 7 '17 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.