0

I was watching this video about the CAP theorem, where the author explains well the trade-offs of distributed systems. However I disagree with the CAP theorem in the following aspect. Given the picture below:

enter image description here

Whenever there is a partition, in other words, whenever a slave loses its connection to the master, this slave immediately becomes unavailable. So you will say: You are choosing consistency over availability. And I will say NO!. My distributed system is still highly available because there are many other backup/redundant slave nodes that the client can failover to. So I'm keeping my consistency and I'm keeping my availability in the system. A failing slave node is immediately (and automatically) taking offline and the client is redirected to another slave node for reads.

Then you might say: now what happens if the master node dies, or if you have a partition where two master nodes are active? And the answer is simple: Your system must NEVER allow two master nodes to be active. Your system must always have one and only one master node with as many backup master nodes as you want, however all the backup master nodes will be inactive (i.e. not accepting writes and just building redundant state).

The only trade-off of such a system, because nothing is perfect: It will need human intervention for the case of a dying / bad state master, so that the active master can be shutdown by a human and guaranteed to be dead while the operator turns on (manually) one of the backup masters to take over write requests.

I've been thinking for a long time on how to eliminate this human intervention, but I don't think it is possible due to the fact that a machine cannot reliably determine the state of another machine in a distributed system. A human needs to make this decision and manually pull the plug to kill it.

Wouldn't this simple trade-off (human operator for the rare cases when the master is dying) beat the CAP theorem?

  • 4
    That seems like a fairly significant trade off... – Telastyn Aug 22 '16 at 13:43
  • Depends on how often the master node dies, agree? – Pika Sucar Aug 22 '16 at 13:46
  • 2
    Who is going to wake up the human? Sitting around waiting for a master node to die would be a very boring job - easy to fall asleep. – Dan Pichelman Aug 22 '16 at 13:48
  • 4
    Your scenario of a slave loosing connection to the master is too narrow to represent the full range of possible network partitions. Consider, just hypothetically, that the master & slave are in different time-zones and some internet router(s) somewhere fail, and there are some clients who can get to the master but not slave and some clients who can get to the slave but master. – Erik Eidt Aug 22 '16 at 14:34
  • 4
    How does the human reliably determine the state of another machine? – Erik Eidt Aug 22 '16 at 14:34
9

whenever a slave loses its connection to the master, this slave immediately becomes unavailable

That is not necessarily true. The CAP argument assumes that when the network is partitioned, there may be clients on both sides of the partition.

...So I'm keeping my consistency and I'm keeping my availability in the system.

The CAP argument also assumes that clients on both sides of the partition want to update the database. If you don't allow them to do that while the partition exists, then the database is not available to all clients for writing. If you do allow them to do that while the network is partitioned, then the database is not consistent because nodes on opposite sides of the partition now have different data.

It's not rocket science.

Your system must NEVER allow two master nodes to be active.

How shall nodes that can't communicate with one another agree upon which one is the master?

If you don't allow updates to a node that can't talk to a master, then you have given up availability again.

It will need human intervention for the case of a dying / bad state master

That would be unacceptable in many of today's large-scale enterprise systems.

A human needs to make this decision and manually pull the plug to kill it.

In the fully general case, maybe so, but if there are any rules that you would write down to guide a new employee in how to make that decision, then you could write those same rules in a computer program that would react much faster than a human operator ever could do.

  • Because you have one and only one master, a network partition means that half of the nodes will have to die, because they are on the side that cannot see the master. Nodes will not vote for the master. The master is determined by a human. Large-scale enterprise systems DO HAVE 24/7 monitoring. – Pika Sucar Aug 22 '16 at 13:52
  • 1
    @PikaSucar If clients on the side of the network that can not see the master are not allowed to update the database, then the database is not available to those clients. CAP assumes that there are clients on both sides of the partition, and that they want to update the database. If that's not true, then we're not talking about the CAP theorem. – Solomon Slow Aug 22 '16 at 14:00
  • You are correct. But I was not clear when I said the nodes will have to die. They will not die but become unavailable until the new master that they can see takes over. So in the event of a network partition, the nodes in the bad side will temporarily become unavailable until a new master that they can see is chosen by the operator. – Pika Sucar Aug 22 '16 at 14:04
  • 2
    @PikaSucar Sounds a lot of like choosing consistency over availability – JimmyJames Aug 22 '16 at 14:10
  • @JimmyJames Also, You are forgetting that your external clients can be re-routed to different nodes for updates, in other words, if the network got partitioned, an incoming external client will have to be redirected to active/live nodes on the good side of the network. That's how load-balance/fail-over works. – Pika Sucar Aug 22 '16 at 14:11
0

You are right. In a distributed system so disperse a human must make the decisions, due to network latency.

But in a simpler architecture, where nodes are close together, a dedicated network interface with a point-to-point cable can be used for heartbeat-checking each other.

Then a high-availability layer in each node can monitor each other.

I administer such a setup with PostgreSQL backends and a Pgpool II in each node monitoring each other with a failover script that decides who the master will be. The failover script also decides which node is assigned the floating IP address end-users know.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.