In a nutshell, should we design death into our programs, processes, and threads at a low level, for the good of the overall system?
Failures happen. Processes die. We plan for disaster and occasionally recover from it. But we rarely design and implement unpredictable program death. We hope that our services' uptimes are as long as we care to keep them running.
A macro-example of this concept is Netflix's Chaos Monkey, which randomly terminates AWS instances in some scenarios. They claim that this has helped them discover problems and build more redundant systems.
What I'm talking about is lower level. The idea is for traditionally long-running processes to randomly exit. This should force redundancy into the design and ultimately produce more resilient systems.
Does this concept already have a name? Is it already being used in the industry?
Based on the comments and answers, I'm afraid I wasn't clear in my question. For clarity:
- yes, I do mean randomly,
- yes, I do mean in production, and
- no, not just for testing.
To explain, I'd like to draw an analogy to multicellular organisms.
In nature, organisms consist of many cells. The cells fork themselves to create redundancy, and they eventually die. But there should always be enough cells of the right kinds for the organism to function. This highly redundant system also facilitates healing when injured. The cells die so the organism lives.
Incorporating random death into a program would force the greater system to adopt redundancy strategies to remain viable. Would these same strategies help the system remain stable in the face of other kinds of unpredictable failure?
And, if anyone has tried this, what is it called? I'd like to read more about it if it already exists.