4

Is it wise to use kafka as the 'source of truth' for mission critical data?

The setup is:

  • kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) hydrated from kafka
  • Kafka configured for durability (infinite topic retention, 3+ replication factor etc)
  • architecture follows CQRS pattern (writes to kafka, reads from the caches)
  • architecture allows for eventual consistency between reads & writes

We are not allowed to lose data in any circumstances

In theory the replication should guarantee durability & resiliency. Confluent themselves encourage the above pattern.

The only flaws I can think of, are:

  • cache blows up and needs to be rehydrated from scratch -> query
  • broker disk gets wiped/corrupted -> kafka rebalance, resulting in prolonged downtime if topics contain mountains of data

Has anyone run and battle tested this kind of setup in production? I.e. encountered disk corruption, brokers going down, but still retain data?

My gut feel is, this is a bad setup, because kafka wasn't originally designed for RMDBS levels of durability, but can't point to a concrete reason why this would be the case.

  • kafka.apache.org/documentation/#design_filesystem implies that data is not fsync()'d to disk (let's pretend that's a strong synchronous guarantee that data is written to the hard disk). This implies to me that it doesn't offer RDBMS levels of durability. But I'm not sure... In: kafka.apache.org/intro under "Kafka as a storage system", it says: "Kafka allows producers to wait on acknowledgement so that a write isn't considered complete until it is fully replicated and guaranteed to persist even if the server written to fails." So it's "unlikely" to lose data :) – Jake_Howard Mar 28 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.