2

I need to stream changes from my MongoDB instance to Elasticsearch. I also need to perform complex transformations on changed documents. After doing some research I narrowed my options down to 2:

  1. MongoDB native change streams
  2. Using a Kafka connector like Debezium to capture MongoDB and stream them to Kafka, then have a consumer subscribed to a Debezium topic

In both cases I can perform complex document transformations because I can use programming language of my choice once a document is consumed, then use an Elasticsearch client in order to insert the document to Elasticsearch. However change streams option seems much more simple. So I'm wondering which use cases justify using a more complex option (Kafka connector) or maybe I'm missing something.

My MongoDB setup is a replica set with a single primary node (it used to be a standalone instance which I converted to replica set solely to enable oplog collection so that MongoDB can track changes to documents).

3
  • Why do you think there's any advantage? Given your current situation, do you see any benefit in implementing Debezium over MongoDB native change streams? Do you need to decouple the streaming source from the DB? You don't compare 2 hammers and think "this is better". You compare two hammers and think "this hammer is more adequate for the job than this other. Atm".
    – Laiv
    Jul 31 at 10:45
  • I guess I’m curious what is the use case when Debezium should be used over change streams.
    – Yos
    Jul 31 at 10:57
  • 1
    When you don't want consumers to be aware of the DB being tracked by Debezium. Or when you need to scale up/out the streaming without modifying the DB setup.
    – Laiv
    Jul 31 at 11:00
1

Well if I understand the situation sufficiently, then I would say that the solution is neither one, but rather using them BOTH together.

This is a solution which is very commonly used all across the industry, to address problems relating to Real-Time Data Aggregation and Analytics. The architecture can be conceptualized as shown below. You have all of your individual data sources over on the left, which are aggregated into your event-streaming platform, which utilizes mongo DB to store the results after your filtering is applied, thus ensuring that you have eventual consistency in the DB. While simultaneously allowing the Kafka platform to handle filtering and analytics in a real-time, and idempotent fashion.

Below the architecture diagram I have linked the reference to it's source, an excellent post, discussing the solution to a similar scenario as your own, for a more in-depth analysis of the potential solution.


MongoDb and Kafka Real-Time Data Streaming Architecture

(Image sourced from NoSQL Data Streaming with MongoDB & Kafka)

1

You have a classic trade off between a simpler narrower-purpose solution and a more complex, but more general purpose, solution. Change streams work for streaming changes from MongoDB, but what if you have sources other than MongoDB in the future? Also, I don't know how change streams scale, but kafka topics are designed from the ground up to partition for horizontal scaling, if you find you need to do your processing faster.

It doesn't instantly follow that you should use a more complex solution because you "might need it" in the future, but you should weigh the likelihood of that against the extra up front work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.