1

We are starting a journey in our company to use Kafka as our event backbone for both streaming data and for an event driven architecture. I foresee a pattern where we would want to consume events off Kafka and update a database, and also expose REST APIs to retrieve data from that database.

I had assumed a Kafka consumer would be its own standalone service with a separate API service to expose the REST APIs, but I'm starting to question this. Is it a common pattern to have a Kafka consumer also be a REST API service? Some things that doesn't sit well with me doing this are:

  • A kafka consumer and a restful service are two separate responsibilities that we're mushing together
  • I find it weird that we'd be bootstrapping an ExpressJS application and then polling Kafka to consume the messages.
  • If either service needs to be scaled we're also scaling the other for no good reason, and seeing that scaling Kafka consumers is limited by the topic's partition it seems like a waste.
Improve this question
1
  • I think this depends on a lot of factor. For example can the consumer access the rest api database? How much traffic do you expect on the api? How much messages per second do you expect from kafka. If you have only a few hundred per second it could be ok to start simple and put them together. According to your first argument this also depends on the point of view. Technical speaking yes, domain specific not necessary.
    – Darem
    Commented Jul 7, 2021 at 13:27

1 Answer 1

Reset to default
1

I have a similar setup and in general separate out the data pipeline (Kafka -> DB) from the APIs used to access that data (HTTP -> DB) based on the Single Responsibility Principle. In general this does make things easier in terms of scalability, access control, testing, etc. but at the same time it creates more components to worry about.

In my case I already have 50 microservices so adding another few more microservices to split Kafka from the REST API is no big deal but if I only had a single service then I would probably combine it and keep it simple according to the principle KISS.

Also as a bonus some companies are ditching the traditional SQL/noSQL DB and using Kafka as their DB.

Improve this answer
1
  • Thanks, your reasonings sound logical. I'm leaning towards separating the two for simplicity but also cause the logs from using KafkaJS SDK client spits out a lot of logs any time a consumer joins the consumer group.
    – n00b
    Commented Jul 12, 2021 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.