Can Software Kill? (you can read a PDF version of the article with pretty pictures). It is a horribly scary story about the threat of bad design in medical devices. There are both historical and current stories of deaths and accidents caused by these machines (Therac-25 accidents of a Canadian design machine, Summer of 2003 power outage, etc.). The article has some very strong recommendations including:
- Certify Programmer
- Certify Teams
- Check, Recheck and Check Again
- Raise the Bar
- Test, Retest – and establish a seal of Safety
- Don’t Buy Problems
This is not a cautionary tale for medical technicians, even though they can find themselves fighting to stay out of jail if they misunderstand or misuse technology. This also is not a tale of how human beings can be injured or worse by poorly designed or poorly explained software, although there are plenty of examples to make the point. This is a warning for any creator of computer programs: that software quality matters, that applications must be foolproof, and that – whether embedded in the engine of a car, a robotic arm in a factory or a healing device in a hospital-poorly deployed code can kill.
This story about the responsibility of software companies and risks of poorly designed software and user interfaces reminded me of the horrible stories in Stephen Casey’s Set Phasers on Stun and Edward Tenner’s Why Things Bite Back. Both books look at the role of human error and poor design of complex systems. They are mandatory reading for all UI/UE/UX/HF/HCI/Usability students. We’ve had it easy these past couple of years, with the focus on consumer e-commerce sites (Ebay, Amazon, etc.) and web tools (Yahoo!, Google, etc.) it is pretty hard to imagine a user experience that might kill our users. However with a possible manned mission to Mars, DARPA Grand Chanllenge and their potential impact on peoples’ lives. The need to revert to traditional human factors engineering practices may be greater than ever, because how do you calculate the ROI of only killing 2 of the three NASA astronauts or the value of only killing a single bystander with an unmanned vehicle?