4

I have a in-house messaging system, similar to a message broker. We have one master message broker and one slave message broker. A message broker just receives messages and sends them to all nodes. The slave is acting as a node, receiving messages from master and building state so it can take over in case of master failure.

Now my problem is: how can I detect, if possible and without human intervention, that the master is dead!? The master may look dead, and the slave might be tempted to take over, but then you might end up in the situation of two masters in your system.

I'm trying to understand how clustering systems implement master-dead detection. Until now it looks like a human has to manually kill the master and turn on a slave, but it would be much more preferable for this process to be automatic.

Improve this question
4
  • 1
    Have a look at heartbeat.
    – Markus Malkusch
    May 28, 2016 at 18:11
  • 3
    With two participants a net-split is indistinguishable from a dead master. You need more than two participants so you can use some kind of majority vote to determine the master. I believe Paxos is the standard approach to distributed consensus.
    – CodesInChaos
    May 28, 2016 at 18:25
  • I believe (can be wrong) for the simple case of ONE master and ONE slave, we don't need the complexity of Paxos.
    – Pika Sucar
    May 28, 2016 at 22:30
  • In addition to the answers: sometimes it's way easier to detect that the broker is down from the point of the services. So instead of checking each other monitor the full app and detect that no messages are coming in anymore. Then from that point decide to block the old master, promote the slave to master and report the incident. If your really need a full recovery and full restore of the master you can do that. But alternatives like initiating a new slave might be easier. Also it allows you to investigate the crashed master to solve the real cause because it becomes untouched.
    – Luc Franken
    Jul 18, 2016 at 19:03

2 Answers 2

Reset to default
3

I'd suggest to define criteria of what "dead" means, then periodically poll for the "dead" condition and perform the swing over. Perhaps "dead" gets defined as "hasn't sent any messages to any of the nodes in X seconds". Whatever decision tree a human currently follows to ascertain whether or not to flip service. It may be 1 condition, 10, or dozens. How well the logic is defined will control how accurately it detects "dead" and fails over.

Also, the swing over process should include informing the "dead" master that it has been declared as dead and should not perform any "master" type of operations. With one exception -- you might want it to retry any messages that had been passed while it was master but did not go.

Or, if the client code is under your control, have the clients manage re-trying failed messages. You need something in place to prevent messages falling through the cracks.

Would be a good idea to also have the dead master, if it comes back online, to come online as a secondary..... and have the "deadness" detector now polling the new master and ready to fail back to the original master if that master dies and the original master is up.

Improve this answer
0

Look at CAP Theorem. If you want Partition Tolerance (i.e. master seems dead but not really), you must sacrifice either Consistency (i.e. master consensus) or Availability (i.e. automatic handling of downed server without downtime). You can't have all three.

Also as CodeInChaos said in comment, with only one master and one slave, you cannot distinguish a dead master from a partitioned network. To be able to detect and recover from network partitioning without massive consistency issue, you need at least three replicas.

If you're willing to sacrifice Consistency, then with two replicas, you can have the slave take over and declare it's a new master after incrementing a switch over version number. Everything that the old master do will be done with the old switch over version number, and when the master and slave reconnects, anything that the slave had not acknowledged prior to the network partition will need to be discarded.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.