6

I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform.

I was wondering if it is common and/or considered a good practice to use the API Management as a gateway to the Kafka topics. The benefits could be analytics, caching, rate limiters and improved access control in the gateway layer (possibly others).

However, the communication between microservices is always a main topic in discussions. My main doubts are:

  1. When communicating, should my microservices call the internal APIs directly (via API Management) or should they produce/consume Events directly from Kafka topics? Is it an anti-pattern to have some microservices to expose APIs, but then some changes to the underlying data occur without making use of this API (i.e. directly in Kafka)?
  2. Since I'm using Event Sourcing and all microservices have access to this single source of truth, do I even need that much communication between microservices? I could duplicate the data that I need in local sources and only worry about communicating for transactions where several other microservices need to take action (e.g. a payment pipeline). Is this a good practice?
Improve this question
1
  • Not an answer to your question, but there is a concern with using Kafka for Event Sourcing. ES usually entails that your services need the ability to reconstitute the state of a single aggregate by reading the events for that aggregate. The most feasible way of doing that efficiently on Kafka would be to create a separate topic for each aggregate. However, topics are heavyweight objects - you don't want to have millions of them. Event Sourcing on Kafka works if you have relatively few (1000's), long-lived aggregates, but it is problematic if you have large amounts of aggregates.
    – Allard
    Commented Dec 7, 2017 at 15:38

1 Answer 1

Reset to default
1

Generally speaking, the relation between event streaming and APIs is still a controversial topic and I don't think there is one right answer for such a question. However, for each specific use case, we should be able to find a suitable architect based on many factors like performance, cost, security ...etc.

Before answering your questions, let me clarify some points:

  1. Kafka provides a highly scalable and available solution for storing/streaming events logs. Therefore, it doesn't need caching or rate-limiting. In fact, rate-limiting shouldn't be a requirement for streaming solutions since you need to consume all the events in the topics you subscribed for. You can filter what you are consuming but that will be done from the consumer side.
  2. The main difference between APIs calls and events streaming is the communication protocol. APIs mostly use synchronous communications like HTTP whereas streaming uses asynchronous communications.
  3. In the world of microservices, synchronous communication should be avoided whenever possible (especially for internal services communications).
  4. For external subscribers, Kafka has some solutions like Confluent REST Proxy to expose Kafka topics as HTTP APIs.

Back to your questions:

  1. It is possible to have some scenarios where you need to expose the same data through streams and APIs. Steams return the logs/history while APIs can return the final state of the same aggregate. However, it is more efficient to use events streams whenever possible. It will remove the dependency between the services, use asynchronous calls, and reduce traffic.
  2. That is correct. But it is worth mentioning that you should make sure the duplication is only for data from different bounded contexts and that they are eventually consistent. Also, it is worth mentioning that even Saga Patterns like payments can be handled asynchronously using event sourcing.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.