51

An airplane, as opposed to, for example, a website, is a system where any failure in certain systems is completely unacceptable, since errors in e.g. flight monitoring can cause the autopilot to malfunction and do a dive. Obviously, this doesn't happen since the brilliant engineers at Boeing and Airbus have checks in the autopilot to make sure it doesn't suddenly decide a dive is a perfectly acceptable and safe maneuver. Or perhaps the computer crashes, and the pilots in the newer fly-by-wire aircraft can no longer actually fly the plane. Of course, there are various safety procedures and redundancies built into these systems to prevent a crash (of both the software and the aircraft).

However, on the other hand, it's quite obvious that software isn't perfect—both open source and closed source software do crash regularly, and only the simplest "Hello World" program doesn't fail. How can the engineers who design the software systems in the aeronautic, medical, and other life-or-death industries manage to test their software so that it doesn't fail (and if it does fail, at least fail gracefully)?

I'm desperately hoping that you're not all going to go: "Oh, I work for Boeing/Airbus/(some other company) and it's not! Have fun on your next flight/hospital visit."

9
  • 8
    Considering the case of the Therac25 and the Patriot battery that didn't engage, obviously not well enough. Commented Jan 29, 2011 at 1:04
  • @Loren Well, I have no doubt that there aren't exceptions. On the other hand, I've never read of a Airbus A320 (a fly-by-wire aircraft) to ever experience a significant software error that resulted in near injury/injury/death, and there have been over 4000 made.
    – waiwai933
    Commented Jan 29, 2011 at 1:14
  • 3
    "Hello World" may also fail. lol
    – xandy
    Commented Jan 29, 2011 at 3:12
  • 1
    @waiwai - actually, that did happen a year or so ago - a faulty sensor indicated that the plane was climbing too steeply and about to stall. The computer's attempt to return to level flight was actually a dive. No crash, but there were injuries/damage from passengers and loose objects getting thrown around the cabin. Commented Jan 29, 2011 at 5:06
  • 6
    Don't they use death-row inmates with pilots licenses?
    – JeffO
    Commented Jan 29, 2011 at 13:41

11 Answers 11

30

I've done a lot of work in industrial controls. It doesn't have to be in a glorious industry like aerospace. Almost every industrial machine has enough potential energy to cause serious injury or death. I have been around when people were injured. If you spend most of your time at a desk in an office, you'd probably be surprised at how dangerous most factory jobs can be (and certainly were until recently). Now we have better methods of machine safeguarding. Here's how it works in practice (though it varies from jurisdiction to jurisdiction):

There are OSHA standards in the US, and similar (usually more strict) guidelines in the EU. These generally start by requiring you to do an analysis of the risk. This means you make a list of all hazards and then categorize those hazards, taking into account things like how often a person would be exposed to the risk, how easy is it to avoid the risk (depends on speed, etc.), and what is the severity of the result (cut, amputation, death, etc.).

A lot of the analysis has to do with guarding hazards. If you put a big cage around your machine and bolt it up tight, then your machine is considered safe if the components of the machine can't breach the guarding. If you need a tool to get in, that's considered a maintenance task, and maintenance people are supposed to be trained on how to safely work on a machine. In reality, however, most machines need regular interaction with operators so we have to put access doors in the guarding, or light curtains, etc. Those doors and light curtains need to be monitored and the power to hazards that the operator is exposing themselves to has to be shut off in a "control reliable" way. This can be complicated if you have a machine that allows full body entry (can a person lock themselves inside?) or with potential energy other than electrical (is there a spinning component that needs to come to rest before the guard can be opened, or is there a vertical ram that needs to be locked in position before the door can be unlocked?)

Based on that analysis, the risk are put into various categories. A common classification scale is Category 1 to Category 4 (based on the EN 954-1 standard). Based on those categories, you are legally required to provide a certain level of machine guarding and safety.

Category 4, for instance, requires that:

A single fault in each of these parts does not cause the loss of the safety function.

The single fault is detected with or before the next request to the safety function, or if this is not possible, an accumulation of faults may not cause the loss of the safety function.

This can be difficult to achieve in practice, but is made simpler by the availability of standard components that are certified to Category 4. For instance, one common component in these systems is a Safety Relay. These are more than just mechanical relays:

  • They are designed to monitor dual redundant input channels, so if you have a sensor that detects a fault condition (like a guard door open), it typically has two contacts with redundant circuits. The relay monitors both channels, and if either one opens, it drops out power to your actuators, but if they both don't drop out at the same time, then it enters a fault condition and the machine can't be restarted until repaired.
  • The relay also uses electrical pulses on those lines and uses those signals to monitor for crossed or shorted wires, so it can detect a wiring fault.
  • On the output side, it uses a set of dual circuits for driving the output coils, so if one faults into the "on" condition, the other should prevent the output from being energized. Additionally these are monitored and if a fault is detected, it prevents operation. The coils themselves are actually dual force guided relays meaning redundant physical relays on the output, plus guaranteeing that the contacts on each single relay are physically linked together so that one contact out of, say 4, can't be stuck by itself. These are also monitored.
  • It also includes an input to monitor an auxiliary normally closed contact off the load you're controlling. If it turns off the output, it has to see the normally closed contact engage meaning it validates that it turned off the motor contactor, or whatever it was, before it's allowed to operate into the on condition again.

As you can tell, these are complicated devices. Typical costs are in the $200 to $600 range for each safety relay. Obviously there's software in these devices. In order to get your safety relay certified, you typically have to follow a design like this:

  • Two redundant processors, typically sourced from different vendors, based on different designs.
  • The code running on each processor has to be developed by two teams working in isolated conditions. This prevents a single software bug from being a single point of failure.
  • The output of both processors has to agree or else the safety relay faults.

Once you design your safety system for your machine, using safety rated components, then you have to get the design reviewed and stamped by a Professional Engineer. Then you build the machine. Then the P.Eng. will review the construction of the machine making sure it was built to the design. They will document it, and will perform some tests on it to make sure it's working as expected. This is called a pre-start review (PSR) and is not done in every jurisdiction. After the PSR passes, then you're allowed to have an operator run the machine.

In recent years there have been some revolutions in safety systems. For a while nobody trusted transmitting safety data over a network, so what's typically called "distributed I/O systems" like DeviceNET and EtherCAT were not allowed in the safety part of the system. However, recent protocols now allow safety devices to run over these industrial networks. The protocols make use of time-stamped messages, and dual redundant processing on both ends of the connection.

Safety relays are slowly going the way of the dodo bird, replaced by more complicated Safety PLCs, which are like a way to build the safety logic in a function block diagram language. Again, these safety PLCs use redundant everything. When the program is approved, before the machine is put into service, the P.Eng. will stamp the program and the program/PLC will be locked with a password. It also takes a hash of the program and that hash is recorded in the documentation (that's what the P.Eng. is really stamping).

Now once you've designed your safety system, the logic you write to control the machine itself can be very seat-of-your-pants. Programmers frequently crash machines causing thousands of dollars of damage, but at least nobody's going to be injured.

20

There's a serious move towards formal verification rather than random functional testing.

Government agencies like NASA and some defense organizations are spending more and more on these technologies.

They're still a PITA for the average programmer, but they're often more effective at testing critical systems.

There's also a tendency to try out more techniques from academia, for things like validating multithreaded code.

7
  • 6
    Having written ground support software for NASA mission control, and seen snippets of old and new flight code, there isn't such an emphasis. Ok, due to the increase in sheer quantity, maybe NASA is spending more on QC than ever before, but the attention focused on each application is far lower than it was when the space program was young. There's still some attention lavished on safety critical things, but mission critical just needs a less-than-comprehensive test suite, and even the safety critical verification appeared to have declined over time.
    – Ben Voigt
    Commented Jan 29, 2011 at 4:24
  • 7
    Please note that I'm not disagreeing that formal verification can be very effective and is essential to producing the highest level of reliability. It's just that managers have learned how much it costs and, faced with more and more complex software, reason that they can't spend that much per requirement and per line of code anymore. The right approach would be to partition the system and keep the critical parts small, but I didn't see that done effectively, with the result that management declared the whole formal verification process prohibitively expensive.
    – Ben Voigt
    Commented Jan 29, 2011 at 4:27
  • 2
    @Ben Voight: If I was an astronaut, I would be highly alarmed by your report.
    – Orbling
    Commented Jan 30, 2011 at 2:11
  • 1
    @Orbling: The astronauts probably ought to throw some weight around the problem, but they're a group of extreme risk takers to begin with. There's a point at which you've reduced the risks you can control to an order of magnitude less than those you can't, and it's not very effective to keep spending money, and it's debatable whether the methods I saw in use got to that point. Some highly placed managers certainly believed that they did.
    – Ben Voigt
    Commented Jan 30, 2011 at 5:51
  • 1
    And it’s sad to think that not many people listened to Dijkstra who had been going on and on about formal verification since the 1960s. As Nietzsche said, “Some men are born posthumously.” Commented Mar 15, 2011 at 0:19
11

It depends on what the software is. For example, in planes there is usually dual-redundant processing for critical systems; in the extreme case there can be 2 different hardware designs used, and two independantly developed pieces of s/w, one to run on each. They both calculate and cross-check each other. This isn't foolproof and is extremely expensive.

When it comes to things like testing of aircraft systems, there are a series of tests done - flight related systems testing takes months, and if you make any change at all a whole series of re-tests are required to be run. This is usually done in a simulator, which might actually be full of real aircraft parts (eg cockpit) with say a simulated engine or similar. As you might imagine, this is also hideously expensive to build. Changes are evaluated against a formal test program, and then run in a real aircraft in test flights. Along the way things like "disturbed functional" tests are run, this is where the changed item is allowed to do its normal stuff and everything is is checked / tested to see that there was no deleterious effect. This also costs a lot of money and can take weeks.

I know of one example where a very simple change to a flight system was required - so simple you'd be stunned at how minor. However the re-test of this would have taken >3 months and cost something like $1M.

When you get into medical, there's a whole series of regulatory hurdles to jump relating not just to testing but to development processes and documentation.

All these fields are a big step up from a places that bashes out a bit of PHP code for a web site. It's slow, painstaking, difficult, boring, tedious, meticulous, and very expensive. Take your normal development/test costs and multiply by about 100 and you are getting close to the mark.

4
  • An example of "2 different hardware designs used, and two independantly developed pieces of s/w, one to run on each" would be interesting. Not disagreeing, just curious. Commented Jan 29, 2011 at 4:53
  • 1
    @Brian: A familiar example for 2 different HW, 2 independently developed SW can be found for example in anti-locking brake system controller. See for example infineon.com/cms/jp/product/applications/automotive/safety/… which uses two different CPU architectures (8-bit and 16-bit) on a single IC.
    – Schedler
    Commented Jan 29, 2011 at 5:31
  • 4
    Three is even better. With two, you only know one of them is wrong. With three, you can take a vote. AFAIK, the Airbus A380 has three flight computers from two manufacturers. Commented Jan 29, 2011 at 11:23
  • Years ago I came across some military head-up displays that were designed this way. My GUESS is that due to cost many of these techniques are no used as much as they once were. Commented Jan 29, 2011 at 23:55
7

For NASA space shuttle software read They Write the Right Stuff. For FDA (US Food and Drug Administration) read read this

3

Since you already got enough great and informative answers, here's my take.

It's simple - the first test is always done on the programmers themselves. It tends to keep bug count low, and ensures that only the quality programmers are kept on the payroll.

0
3

Life-critical software isn't tested to any standard other than the "it seems to work" one, as it's done all over the place.

All the investment goes into either what seemed to work before or for projects to allow non-programmers to produce better software.

p.s. No comments on the first -1, but I'd be happy to take -1 for each reference that counters my statement.

Can I take a +1 for every reference I find to critical software not being well designed or tested? Simson Garfinkel documents ten cases in an article on WIRED.

4
  • This is, unfortunately, all too accurate.
    – Ben Voigt
    Commented Jan 29, 2011 at 4:20
  • Sure, I took you up on that that offer for a -1. Commented Jan 29, 2011 at 4:28
  • @Brian Carlton You didn't provide a reference...
    – Apalala
    Commented Jan 30, 2011 at 1:43
  • What about DO-178, MOPS for GPS... At least were I work testing can take over a year. Note that testing does not ensure that the code is bug-free and conforms to the specification in every possible case.
    – jinawee
    Commented Jan 14, 2019 at 13:51
2

There isn't one answer for all cases. It is up to the individual manufacturer to decide how to design and test their software. But the entire software development process must comply with formal specifications.

For example when creating software for medical devices you must follow the IEC 62304 standard for software for medical devices. (Unfortunately I can only link to wikipedia as it is not free). Pretty much every country in the world requires that this standard has been followed.

How strict these requirements are depends on the risk of harm. E.g. a life support device would have the greatest risk of harm (certain death it the system fails), whereas a system that works with disease diagnostics have a lower risk (possible death if a a terminal disease was not diagnosed correctly if the system fails).

But what this says basically is that there must be a traceability from requirements to the software. You must perform software unit verifications. That doesn't specify what the verification is. Can be unit tests, can be code review. For the higher risk devices you must manually review the interfaces between software units (as far as I understand and remember). And of course lots of other rules. Oh, and you must write a lot of documentation to document your work.

The standard doesn't prohibit agile development, though when reading it it seems like it was written with waterfall development in mind.

I don't know about other areas of software development, such as aviation, trains, cars, etc. But I assume that other similar formal guidelines exists.

1

Many techniques are used, including but not limited to:

  • formal design and/or validation
  • strict coding standards avoiding fancy programming such as dynamic allocation of memory
  • very demanding quality engineers
  • manys static analysis techniques such as code review, fault trees, FMECA, ...

But the number one technique is:

  • TESTING

Software of a spacecraft needs more effort to test than to design and code in the first place.

Aircrafts undergo several years of flight tests where the plane is brought into extreme situations. This tests not only the physical structure but also software.

1

There is an article "Perfect software" by Jack Ganssle on EETimes dated 3/1/2009 12:00 AM EST. A few points from there:

  • "Theoretically, Software is the only component that can be perfect, and this should always be our starting point." - That is said by Jesse Poore. Following his web page you'll find out how perfect software can be made at no much more cost than normal software.
  • There are industrial providers of highly reliable OS's. The article mentioned Mircrium and Green Hills. Following Micrium's web page, there are list of standards for certifications. Those must be the processes and rules the industry follow. That's based on formal validation, but diverted from the theory a lot.

Interestingly, regarding the commercial software, data collected by Capers Jones suggests that "software in general has a defect removal efficiency (the percentage of bugs removed prior to shipping) of 87%. Firmware scores a hugely better 94%." To me, none of these are near perfect. The article an earlier answerer mentioned indicates that NASA space shuttle team achieved a 99% bug removal rate, but the cost is at 35 million per year for about 400k lines of code.

A more interesting article "Software for dependable systems" by the same author on 11/1/2009, seems to be more relevant. It can be summarized like this:

  • As to development process, the industry has followed DO-178B or IEC61508. It emphasize testing with greatest coverage. But little evidence can be found about effectiveness of this.
  • A certification authority, the Committe on Certificably Dependable Software Systems, has published a book titled "Software for Dependable Systems - Sufficient Evidence". It's a good reference.
  • The book, basically asks for three things: [1] Make explicit rules for dependability tests. [2] Tests against the rules, but also make sure the tests are understandable by normal customers or regulators with a magnitude less time than spent by the developers. [3] Be sure it is the experts in software engineering and problem domain are doing the development and test.
  • The supplier of software must commit to a warranty or other remedies for any software failure.
  • Use a safe language, specifically not C. SPARK is suggested.
  • Design for contract approach is one of the strength of SPARK. Design-for-contract is an important technology to embed tests into every function/method call, and always execute it at runtime. More than a simple assert() in old days, the contract may also embeds tests against structural intention and make sure no violation can be slipped in at later times in maintenance cycle when the original developers mostly have moved on to other projects. There are evidences that design for contract has produced very reliable software products. SPARK and its tools can be used to assist generating certification test cases to simplify the certification process.

According to my memory HP practiced design-for-contract almost a decade ago. With a small team, 500k lines of code, only 2 bugs reported after the delivery. Very impressive.

In my view, dependable or perfect software can only be achieved if the cost is not prohibitively high. Frameworks or automations is a must-have.

0

They usually have a hardware interlock that is used as a fail-safe.

E.g. standard evil text box designs for killer robots always come with a kill-switch :P

0

Each industry has its own set of regulatory agencies that have testing and documentation requirements for safety related hardware and software. Consider this PDF from Underwriters Laboratory (UL) that introduces the UL 1998 standard: http://www.ul.com/global/documents/offerings/industries/hightech/software/UL_softwareconformityassessment.pdf

There are references in that document to many other related ones from UL, CSA, and IEC.

Typically, safety related software will have redundant hardware circuits or be required to have other redundant control features to ensure safe operation and safe failure modes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.