Richard G. Epstein, Westchester University of Pennsylvania
Mike Melamed, CWRU 2000
The Case of the Killer Robot is a detailed scenario that combines elements of software engineering and computer ethics.
The scenario consists of fictitious articles that touch on specific issues in software engineering and computer ethics. The articles discuss programs such as programmer psychology, team dynamics, user interfaces, software process models, software testing, the nature of requirements, software theft, and privacy. A major consideration is "when is the software good enough?"
The articles in the scenario begin with the indictment for manslaughter of a programmer who wrote faulty code that caused the death of a robot operator. Slowly, over the course of many articles, students are introduced to factors within the software company that also contributed to the accident. They are shown software development as a social process. It is hoped that students will begin to realize the complexity of the task of building real-world software and to see some of the ethical issues intertwined in that complexity.
This scenario is about 70 pages long and includes some tongue-in-cheek humor.
For permission see author information.
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
At a news conference this afternoon, a ragtag group of programmers who call themselves the Justice for Randy Samuels Committee distributed documents that show Silicon Techtronics obligated itself to deliver robots that would "cause no bodily injury to the human operator." Randy Samuels is the programmer charged with manslaughter in the infamous "killer robot" case.
"We cannot understand how the prosecuting attorney could charge Randy with manslaughter when, in fact, Silicon Techtronics was legally bound to deliver a safe robot to Cybernetics," said committee spokesperson Ruth Witherspoon. "We believe that there is a cover-up going on and that there is some kind of collusion between SiliTech (Silicon Techtronics) management and the prosecuting attorney's office. Michael Waterson was a major contributor to Ms. McMurdock's re-election campaign last year." Michael Waterson is president and CEO of Silicon Techtronics. Jane McMurdock is the prosecuting attorney for the city of Silicon Valley. The Sentinel-Observer has confirmed that Waterson made several large contributions to the McMurdock re-election campaign last fall.
"Randy is being made the scapegoat for a company which had lax quality-control standards, and we are not going to stand for it!" Witherspoon shouted in an emotional statement to reporters. "We believe that politics has entered this case."
The documents distributed by the Justice for Randy Samuels Committee were portions of what is called a "requirements document." According to Witherspoon and other committee members, this document proves that Samuels was not legally responsible for the death of Bart Matthews, the unfortunate robot operator who was killed by a Silicon Techtronics robot at Cybernetics Inc., in Silicon Heights, last May. The requirements document amounts to a contract between Silicon Techtronics and Cybernetics Inc. It spells out in complete detail the functionality of the Robbie CX30 robot Silicon Techtronics promised to deliver to Cybernetics.
According to Witherspoon, the Robbie CX30 robot was designed to be an "intelligent" robot capable of operating in a variety of industrial settings. Separate requirements documents were necessary for each corporate customer because the Robbie CX30 was not an "off-the-shelf" robot but a robot that needed to be programmed differently for each application. However, all requirements documents for the Robbie CX30 project, including the agreement between Silicon Techtronics and Cybernetics, contain the following important statements:
"The robot will be safe to operate and even under exceptional conditions (see Section 5.2) the robot will cause no bodily injury to the human operator . . . . In the event of the exceptional conditions which potentially contain the risk of bodily injury (see Section 5.2.4 and all of its subsections), the human operator will be able to enter a sequence of command codes, as described in the relevant sections of the functional specification (see Section 3.5.2), which will arrest robot motion long before bodily injury can actually occur."
Exceptional conditions include unusual events such as bizarre data from the robot sensors, erratic or violent robot motion or operator error. It was exactly such an exceptional condition that led to the death of Bart Matthews. These passages were extracted from the portion of the requirements document that deals with "non-functional requirements." The non-functional requirements present in complete detail the constraints under which the robot will operate. For example, the requirement that the robot be incapable of harming its human operator is one such constraint, and Silicon Techtronics, according to Witherspoon, was legally obligated to satisfy this constraint.
Elsewhere, the "functional requirements" portion of the requirements document covers, again in complete detail, the behavior of the robot and its interaction with its environment and its human operator. In particular, the functional requirements specified the behavior of the robot under each and every anticipated exceptional condition. Exceptional conditions that require operator intervention cause an error message to be generated at the operator console. In her statement to reporters, Witherspoon explained that Bart Matthews was killed when exceptional condition 220.127.116.11 arose. This involved an exceptionally violent and unpredictable robot arm motion. This is a condition that requires operator intervention, namely the entering of the command codes mentioned in the document, but apparently Bart Matthews became confused and could not enter the codes successfully. Silicon Valley Police confirm that when Bart Matthews was killed, the reference manual at his console was opened to the page of the index which contained entries for "errors."
"Although Randy Samuels' program was in error -- he did misinterpret the robot dynamics formulas, as reported in the media -- exceptional condition 18.104.22.168 was designed to protect against just this sort of contingency," Witherspoon told reporters. "The robot motion values generated by Randy's program correctly set off this exceptional condition, and the robot operator received due warning that something was wrong."
Witherspoon claimed that she has a signed affidavit from another Cybernetics robot operator to the effect that the training sessions offered by Silicon Techtronics never mentioned this nor many other exceptional conditions . According to Witherspoon, the robot operator has sworn that neither she nor any other robot operator was ever told that the robot arm could oscillate violently. Witherspoon quoted the affidavit at the news conference: "Neither I nor Bart Matthews was ever trained to handle this sort of exceptional condition. I doubt that Bart Matthews had any idea what he was supposed to do when the computer screen started flashing the error message on the screen".
Witherspoon then quoted sections of the requirements document that obligated Silicon Techtronics, referred to as "the vendor," to adequately train robot operators: "The vendor shall provide forty (40) hours of operator training. This training shall cover all aspects of robot operation, including exhaustive coverage of the safety procedures which must be followed in the case of exceptional conditions which potentially contain the risk of bodily injury . . . . The vendor shall provide and administer appropriate test instruments which shall be used to certify sufficient operator understanding of robot console operations and safety procedures. Only employees of the customer who have passed this test shall be allowed to operate the Robbie CX30 robot in an actual industrial setting . . . . The reference manual shall provide clear instructions for operator intervention in all exceptional situations, especially and including those which potentially contain the risk of bodily injury."
According to Witherspoon, sworn affidavits from several robot operators at Cybernetics state that only one work day of approximately eight hours was spent in operator training. Furthermore, almost no time was spent discussing potentially dangerous exceptional conditions.
"The written test developed by Silicon Techtronics to certify a robot operator was considered a joke by Cybernetics employees," Witherspoon asserted. "Silicon Techtronics obviously did not give much thought to the training and testing procedures mandated by the requirements document, according to the evidence in our possession."