As I understand, consensus is the mechanism that will ensure the ledger is kept in sync across nodes. Is that correct? Also its nothing new and wasn't something that was invented with the advent of blockchain. It is an integral part of distributed databases and has been around for decades. Many algorithms exist to drive consensus - Paxos, Raft, (P)BFT etc.
I am trying to understand if Hyperledger Fabric really comes with consensus protocol/mechanism. The docsdocs say the orderer establishes consensus and currently its implemented via Apache Kafka.
But when I think about it a little deeper, Apache Kafka is not something that provides consensus. It is a pub-sub service. I have never heard of Kafka in the context of consensus. When people talk of consensus they mention terms like BFT, Paxos, Raft etc. not Kafka. Kafka is powered by Zookeeper which indeed is a technology to provide consensus. But the consensus Zookeeper will provide is among the Kafka nodes, not the nodes of the blockchain!
Indeed quoting thisthis paper p.3 Section 1:
An ordering service atomically broadcasts state updates to peers and establishes consensus on the order of transactions
I agree with this - consensus is established on the order of transactions. But where is the consensus to ensure the ledger is in sync across peer nodes? Is that not needed? And if everything is good with Kafkaso then why are Fabric developers working to implement Raft consensus [1] - what is the reason and motivation behind it?