We are starting a journey in our company to use Kafka as our event backbone for both streaming data and for an event driven architecture. I foresee a pattern where we would want to consume events off Kafka and update a database, and also expose REST APIs to retrieve data from that database.
I had assumed a Kafka consumer would be its own standalone service with a separate API service to expose the REST APIs, but I'm starting to question this. Is it a common pattern to have a Kafka consumer also be a REST API service? Some things that doesn't sit well with me doing this are:
- A kafka consumer and a restful service are two separate responsibilities that we're mushing together
- I find it weird that we'd be bootstrapping an ExpressJS application and then polling Kafka to consume the messages.
- If either service needs to be scaled we're also scaling the other for no good reason, and seeing that scaling Kafka consumers is limited by the topic's partition it seems like a waste.