1

Mission-critical software system (such as systems-control software in spacecrafts) often employ multiple redundant software modules developed by different teams (sometimes using different programming languages), to same interface and behavioural specifications. The idea being that software bugs that surface only during some extreme corner-cases, not caught in pre-production testing should be localized to implementation in only one of the modules, leaving other modules unimpacted since they are developed by different teams using different tools/languages.

However, there are these somewhat less (mission-)critical systems whole malfunction can have significant financial impact, do manage to be fault-tolerant, seemingly using redundant instance of software written in same language, perhaps by same team of developers, which not only ensure high-availability but also fault-tolerance for ongoing transactions, such that fault on one node serving a request can continue seamlessly on another node. In case of hardware fault, I can understand why this could work. However, in case of software fault, why/how does it work ? Isn't it that the same software fault that caused failure of first node, would impact the other node as well ?

Also, are there some design patterns that permit composing a fault tolerant system using components that are intrinsically not fault tolerant ?

Improve this question
4
  • If all the systems were receiving exactly the same input and running exactly the same software on the same hardware, then yes, the same bug that crashed one node would simultaneously crash all the other nodes.
    – Criticizing Israel not allowed
    Commented Mar 5, 2016 at 7:51
  • 2
    If 1 of your cluster-servers has a MTBF of 1 year, a cluster of 365 of them will have a MTBF of 1 day. This is what the cluster-technologies address, rather than being safe of software bugs. As for the last question: That is exactly the philosphy behind erlang/OTP. "Our software systems will have bugs, we accept that, but we won't accept that our systems fail because of that."
    – BitTickler
    Commented Mar 5, 2016 at 16:54
  • @BitTickler, 1 server w/ MTBF 1yr => 365 server cluster w/ MTBF 1day, seems to be counter intuitive to the argument in favor of clusters, no ? Did I misread or misunderstand ? Indeed, I've read about, and dabbled a little in Erlang, and my question is precisely about those design patterns adopted by Erlang, that make it resilient to software faults.
    – bdutta74
    Commented Mar 5, 2016 at 17:15
  • @icarus74: It makes sense to me - more pieces of hardware means a higher chance that one piece of hardware will fail, which means a greater need for fault tolerance to mitigate the risk of hardware failure. Mitigating the risk of software failure is a similar concept with very different solutions (e.g. single "not redundant at all" server with check-pointing).
    – Brendan
    Commented May 4, 2016 at 10:49

2 Answers 2

Reset to default
2

If the fault is within the hardware such as CPU, or with non-ECC memory modules, then having multiple nodes using the same code but different hardware would indeed help.

But if the fault is within the code and is hardware-independent, it will manifest itself on every node if all the nodes share the same code.

From here, you have three choices:

  • Write better software in the first place. Do code reviews, increase testing coverage. This is the technique used by most companies for projects which are not life-critical, but still expected to be reliable.

  • Use formal proof. partially used for some life-critical projects, this approach has a cost which is so high that you can't reasonably use it for ordinary software products.

  • Ask two teams to independently write the same software using different programming languages and different platforms. This is exactly what you are talking about in your question and could be a good compromise between two other approaches.

Improve this answer
2

The software aspect of fault tolerant systems lies in the environment. While running 2 identical systems would see the same software bug being replicated, its often a case that one system will get itself into a state where it goes wrong (eg a thread locking issue, running out of memory etc), moving processing onto a secondary server would not have the same environment and would continue running correctly (or at least until the 1st was fixed/rebooted).

Bugs fall into 2 main categories - there's bugs that show themselves in normal operation and are easily reproducible (eg you click a menu item and it pops an error dialog because of some failure in coding). But the other type of bug appears almost at random because of the way software is so complex and interdependent on other factors. Threading is a prime example: a program can run perfectly for years until one day it suddenly hangs. Investigating that bug can show a deadlock that no-one considered until the day it showed itself.

These are the bugs that are difficult to detect, and are the reason fault-tolerant systems are created. You can restart the system and clear the fault, but restarting takes time and causes downtime. If you have a redundant system ready to take over, you can restart without any downtime. You understand that most systems work correctly for a long time until something occurs. Even a system with a bug will run correctly

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.