Running Apache ZooKeeper (this applies to PAXOS style system just as well) with 2 members - which is not recommended - does not make sense, because if either

  • member1 goes down
  • member2 goes down
  • there's a network partition between member1 and member2

, ZooKeeper will declare business hours to be over, as there's no single sub cluster with a clear majority of members (in this case: > 1). So running ZooKeeper on 2 members makes it less available than running a single member.

Why can't the majority constraint be relaxed so that when the there's a sub cluster of exactly 50% of members, it is considered the active sub cluster if it contains the eldest member? (Eldest as of the just before the partition happened, i.e. when the cluster was member-complete) Who's eldest is information ZooKeeper already keeps track of, so implementation looks easy.

That way, we don't run into the dreaded split-brain scenario, and the ZooKeeper cluster could stay up in 2 of the 3 cases above, improving availability, especially when running 2 members only.

Anything speaking against this?

  • 2
    It is recommended to run ZooKeeper with an odd number of nodes. – JensG Apr 21 '14 at 13:26
  • using the eldest node as the solution only works if the youngest node crashes. The solution for your problem is to bring in a third ZooKeeper server – Michael Shaw Apr 23 '14 at 11:32
  • @Ptolemy To quote my question: ZooKeeper cluster could stay up in 2 of the 3 cases above, improving availability. In other words: Yes, it works if the youngest member crashes, but it also works in case of a network partition - in which case the eldest member would stay alive. That's positive, in terms of availability. What I want to know is if this can be implemented, or if there are other good reasons against it that I'm missing. – Eugen Dück Apr 23 '14 at 11:59
  • @JensG Yes, but does that answer my question? – Eugen Dück Apr 23 '14 at 13:15
  • @EugenDück: To some extent, yes. Your claim was, that a 2 server scenario is actually less available. But a 2 server scenario is not recommended for exactly that reason, and because an odd number makes it easier to find a quorum. So your question is actually an rather academical one, If it would be a fully fledged answer, I would have posted it as such. – JensG Apr 23 '14 at 13:18

The entire point of Zookeeper (as I understand it) is to make restart AFTER the partition goes away simple by making sure that there's only ONE portion (the majority) that was changing during the partition. It can then bring the minority up to speed when they reconnect and everything is running again with no further intervention on your part.

If you let two minority clusters run, then zookeeper has NO IDEA what to do when they reconnect. If you're lucky it won't take you too long to put the mess back together. If you aren't lucky, some incompatible change will have happened on both sides of the partition and you'll be days cleaning up the mess.

I suspect that if you need the minority portions of a cluster to operate independently, then you either need a solution different than Zookeeper, OR perhaps you don't actually want a cluster at all, perhaps you want a bunch of systems that replicate back and forth somehow and just sync data whenever they can.

| improve this answer | |
  • 1
    Correct. One part of the split brain must be chosen as the victim, being reset and resynchronized with the winning part. – JensG Apr 21 '14 at 13:27

It just occurred to me that although the 2 member scenario could indeed be made more available than it is right now, it would still not be more available than the 1 member scenario, and writing is in fact slower for 2 members, because every change needs to be propagated to another member. Only reads from ZooKeeper will improve under heavy load (assuming the members are on different machines), because you can read from any of two machines, rather than just one.

So this change might still be implemented, but choosing 2 members over just 1 member would turn from a no-brainer to a trade-off decision: having to maintain another process, worse write performance in general, with better read scalability for the improved 2 member case. This isn't necessarily bad though, especially if ZooKeeper is used for what it was designed for: scenarios with an overwhelming proportion of reads.

And to address the question: Although it would increase availability, it would increase the 2 member availability only to that of the 1 member case.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.