I need to stream changes from my MongoDB instance to Elasticsearch. I also need to perform complex transformations on changed documents. After doing some research I narrowed my options down to 2:
- MongoDB native change streams
- Using a Kafka connector like Debezium to capture MongoDB and stream them to Kafka, then have a consumer subscribed to a Debezium topic
In both cases I can perform complex document transformations because I can use programming language of my choice once a document is consumed, then use an Elasticsearch client in order to insert the document to Elasticsearch. However change streams option seems much more simple. So I'm wondering which use cases justify using a more complex option (Kafka connector) or maybe I'm missing something.
My MongoDB setup is a replica set with a single primary node (it used to be a standalone instance which I converted to replica set solely to enable oplog collection so that MongoDB can track changes to documents).