Richard G. Epstein, Westchester University of Pennsylvania
Mike Melamed, CWRU 2000
The Case of the Killer Robot is a detailed scenario that combines elements of software engineering and computer ethics.
The scenario consists of fictitious articles that touch on specific issues in software engineering and computer ethics. The articles discuss programs such as programmer psychology, team dynamics, user interfaces, software process models, software testing, the nature of requirements, software theft, and privacy. A major consideration is "when is the software good enough?"
The articles in the scenario begin with the indictment for manslaughter of a programmer who wrote faulty code that caused the death of a robot operator. Slowly, over the course of many articles, students are introduced to factors within the software company that also contributed to the accident. They are shown software development as a social process. It is hoped that students will begin to realize the complexity of the task of building real-world software and to see some of the ethical issues intertwined in that complexity.
This scenario is about 70 pages long and includes some tongue-in-cheek humor.
For permission see author information.
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
The Robbie CX30 industrial robot was supposed to set a new standard for industrial robot intelligence. Unfortunately, one of the first Robbie CX30 robots killed an assembly-line worker, leading to the indictment of one of the robot's software developers, Randy Samuels. This paper propounds the theory that it is the operator-robot interface designer who should be on trial in this case. The Robbie CX30 robot violates nearly every rule of interface design. This paper focuses on how the Robbie CX30 interface violated every one of Shneiderman's eight golden rules.
Back to the Top
On May 17, 1992 a Silicon Techtronics Robbie CX30 industrial robot killed its operator, Bart Matthews, at Cybernetics, Inc., in Silicon Heights, a suburb of Silicon Valley. An investigation into the cause of the accident led authorities to the conclusion that a software module, written and developed by Randy Samuels, a Silicon Techtronics programmer, was responsible for the erratic and violent robot behavior which in turn lead to the death by decapitation of Bart Matthews.1
As an expert in the area of user interfaces,2, 3, 4 I was asked to help police reconstruct the accident. In order to accomplish this, Silicon Techtronics was asked to provide me with a Robbie CX30 simulator which included the complete robot operator console. This allowed me to investigate the robot's behavior without actually risking serious harm. Due to my extensive understanding of user interfaces and human factors I was able to reconstruct the accident with uncanny accuracy. On the basis of this reconstruction, I came to the conclusion that it was the interface design and not the admittedly flawed software which should be viewed as the culprit in this case.
Despite my finding, Prosecuting Attorney Jane McMurdock insisted on pursuing the case against Randy Samuels. I believe that any competent computer scientist, given an opportunity to interact with the Robbie CX30 simulator, would also conclude that the interface designer and not the programmer should be charged with negligence, if not manslaughter.
My evaluation of the Robbie CX30 user interface is based upon Shneiderman's 'eight golden rules' 5. I also used other techniques to evaluate the interface, but those will be published in separate papers. In this section, I offer a brief review of Shneiderman's eight golden rules, a subject which would be more familiar to computer interface experts such as myself as opposed to the robot hackers who read this obscure journal.
The eight golden rules are:
As we shall see below, it is important for a user interface to be consistent on many levels. For example, screen layouts should be consistent from one screen to another. In an environment using a graphical user interface (GUI), this also implies consistency from one application to another.
Frequent users (or power users) may be turned off by overly tedious procedures. The interface should allow those users a less tedious procedure for accomplishing a given task.
Users need to see the consequences of their actions. It can be confusing and disorienting for the user if the computer does not show either that it is processing or has processed a command that the user has entered.
Interacting with a computer is somewhat like a dialogue or conversation. The dialogue for every task should have a beginning, a middle, and an end, and it is important for the user to know when a task is at its end. The user needs to have the feeling that a task has reached closure.
User errors should be designed into the system. To state this another way: no user action should be considered an error beyond the ability of the system to manage. If the user makes a mistake, he or she should receive useful, concise, and clear information about the nature of the mistake, and it should be easy for the user to undo the mistake.
More generally, users must be permitted to undo what they have done, whether it is an error or not.
User satisfaction is high when the user feels that he or she is in control, and user satisfaction is low when the user feels that the computer is in control. Interfaces should be designed to reinforce the feeling that the user is the focus of control in the human-computer interaction.
Human short-term memory is remarkably limited. Psychologists often quote Miller's law to the effect that short-term memory is limited to seven discrete pieces of information. The interface should do everything possible to lessen the user's memory burden. For example, instead of being asked to type in the name of a file to be retrieved, the user might be presented with a list of files currently available.
The Robbie CX30 operator interface violated each and every one of Shneiderman's rules. Several of these violations were directly responsible for the accident that ended in the death of the robot operator.
The robot console was an IBM PS/2 model 55SX with a 80386 processor and an EGA color monitor with 640x480 resolution. The console had a keyboard, but no mouse. The console was embedded in a workstation which included shelves for manuals and an area for taking notes and for reading manuals. However, the reading/writing area was quite a distance from the computer screen, so it was quite awkward and tiresome for the operator to manage any task that required looking something up in the manual and then acting quickly at the console keyboard. The operator's chair was poorly designed and much too high relative to the console and the reading/writing area. This placed much strain on the operator's back and also caused excessive eye strain.
I cannot understand why a sophisticated system such as this would not include a better device for input. One can only conclude that Silicon Techtronics did not have much experience with user interface technology. The requirements document6 specified a menu-driven system, which was a reasonable choice. However, in an application where speed was of the essence, especially when operator safety was at issue, the use of a keyboard for all menu selection tasks was an extremely poor choice, requiring many keystrokes to achieve the same effect which could be achieved almost instantaneously with a mouse. (See the paper by Foley et. al.)7. (Actually, I had most of these ideas before Foley published them, but he beat me to the punch.)
The robot operator interacted with the robot and made an impact on its behavior by making choices in a menu system. The main menu consisted of 20 items -- too many, in my opinion -- and each main menu item had a pull-down submenu associated with it. Some of the submenus contained as many as 20 items -- again, too many. Furthermore, there seemed to be little rhyme or reason as to why the menu items were listed in the order in which they appeared. A functional or alphabetical organization would have been better.
In the pull-down submenus, some items had up to four pop-up menus associated with them. These would appear in sequence as submenu choices were made. Occasionally, a submenu choice would cause a dialogue box to appear at the screen. A dialogue box requires some kind of interaction between the operator and the system to resolve some issue, such as the diameter of the widgets being lowered into the acid bath.
A menu system presents a strict hierarchy of menu choices. On this system, the operator could backtrack up the hierarchy by pressing the escape key. The escape key could also terminate any dialogue. In addition, the use of color in the interface was very unprofessional: There were too many colors in too small a space. The contrasts were glaring and the result, for this reviewer, was severe eye strain in just fifteen minutes. There was also excessive use of screen flashing and silly musical effects when erroneous choices or erroneous inputs were made.
One has to wonder why Silicon Techtronics did not attempt a more sophisticated approach to the interface design. After a careful study of the Robbie CX30 applications domain, I have come to the conclusion that a direct manipulation interface, which literally displayed the robot at the operator console, would have been ideal. The very visual domain that the robot operated within would lend itself naturally to the design of appropriate screen metaphors for that environment, metaphors the operator could easily understand. This would allow the operator to manipulate the robot by manipulating the graphical representation of the robot at the computer console. I have asked one of my doctoral students, Susan Farnsworth, to give up her personal life for the better part of a decade in order to investigate this possibility a bit further.
The Robbie CX30 user interface violated each and every golden rule in multitudinous ways. I shall only discuss a few instances of rule violation in this paper, leaving a more detailed discussion of these violations for future articles and my forthcoming book8. I will emphasize those violations which were relevant to this particular accident.
There were many violations of consistency in the Robbie CX30 user interface. Error messages could appear in almost any color and could be accompanied by almost any kind of musical effect. Error messages could appear almost anywhere at the screen.
When Bart Matthews saw the error message for the exceptional condition that occurred, an exceptional condition that required operator intervention, it was probably the first time he had seen that particular message. In addition, the error message appeared in a green box, without any audio effects. This is the only error message in the entire system which appears in green and without some kind of orchestral accompaniment.
This principle did not appear in any way in the entire interface design. For example, it would have been a good idea to allow frequent users to enter the first letter of a submenu or menu choice to effect a choice, in lieu of requiring the use of the cursor keys and the enter key. The menu selection mechanism in this system must have been quite a mental strain on the operator.
Furthermore, a form of type-ahead should have been supported, which would have allowed a frequent user to enter a sequence of menu choices without having to wait for the actual menus to appear.
There are many cases in which a given sequence of keystrokes represents one holistic idea, one complete task, but the operator is left without the kind of feedback which would confirm that the task has been completed. For example, there was a fairly complicated dialogue necessary to remove a widget from the acid bath. However, upon completion of this dialogue, the robot operator was led into a new, unrelated dialogue, without being informed that the widget removal dialogue had been completed.
There are many cases in which a given sequence of keystrokes represents one holistic idea, one complete task, but the operator is left without the kind of feedback which would confirm that the task has been completed. For example, there is a fairly complicated dialogue which is necessary in order to remove a widget from the acid bath. However, upon completion of this dialogue, the user is led into a new, unrelated dialogue, without being informed that the widget removal dialogue has been completed.
The system seems to be designed to make the user regret any erroneous input. Not only did the system allow numerous opportunities for error, but when an error actually occurred, it was something that was not likely to be repeated for some time. This is because the user interface made recovery from an error a tedious, frustrating and at times infuriating ordeal. Some of the error messages were downright offensive and condescending.
As mentioned in the previous paragraph, the user interface made it very difficult to recover from erroneous inputs. In general, the menu system did allow easy reversal of actions, but this philosophy was not carried through to the design of dialogue boxes or to the handling of exceptional conditions. From a practical (as opposed to theoretical) point of view, most actions were irreversible when the system was in an exceptional state, and this helped lead to the "killer robot" tragedy.
Many of the deficiencies discussed in the previous paragraphs diminished the feeling of an "internal locus of control," for example: not providing feedback, not bringing interactions to closure, and not allowing easy reversal of actions when exceptions arose. All of these things acted to diminish the user's feeling of being in control of the robot. There were many features of this interface that could make the user feel as though there was an enormous gap between the operator console and the robot itself, whereas a good interface design would have made the user interface transparent and would have given the robot operator a feeling of being in direct contact with the robot. In one case, I commanded the robot to move a widget from the acid bath to the drying chamber and it took 20 seconds before the robot seemed to respond. Thus, I did not feel as though I was controlling the robot. The robot's delayed response, along with the lack of informative feedback at the computer screen, made me feel that the robot was an autonomous agent -- an unsettling feeling, to say the least.
A menu driven system is generally good in terms of the memory burden it places upon users. However, there is great variation among particular implementations of menu systems. The Robbie CX30 user interface had very large menus without any obvious internal organization. These placed a great burden upon the operator in terms of memory, and also in terms of scan time, the time it takes the operator to locate a particular menu choice.
In addition, many dialogue boxes required the user to enter part numbers, file names, and other information from the keyboard. This greatly increased the memory burden upon the user. The system could easily have been designed to present the user with part numbers, and so forth, so the user would not have to recall these things from his or her own memory.
Finally, and this is really unforgivable, incredible as it may seem, there was no on-line, context-sensitive help facility! Although I was taken through the training course offered by Silicon Techtronics, I often found myself leafing through the reference manuals in order to find the answer to even the most basic questions, such as: "What does this menu choice mean? What will happen if I make this choice?"
Police photographs of the accident scene are not a pleasant sight. The operator console was splattered with a considerable amount of blood. However, the photographs are of exceptional quality and using blow-up techniques, I was able to ascertain the following important facts about the moment when Bart Matthews was decapitated:
The IBM keyboard contains a calculator pad that can operate in two modes. When the NUM LOCK light is on, it behaves like a calculator. Otherwise, the keys can be used to move the cursor at the screen.
Bloody fingerprints indicate that Bart Matthews was using the calculator pad when he was struck and killed.
This tells us the error situation in force when the tragedy occurred. The error message said, "ROBOT DYNAMICS INTEGRITY ERROR - 45 ".
This one volume of the four volume reference manual was open to the index page that contained the entry "ERRORS/MESSAGES."
This message was displayed in yellow at the bottom of the screen. It read "PLEASE ENTER DYNAMICAL ERROR ROBOT ABORT COMMAND SEQUENCE PROMPTLY!!!"
On the basis of this physical evidence, plus other evidence contained in the system log, and based upon the nature of the error that occurred ("robot dynamics integrity error - 45," an error caused by Randy Samuels' program), I have concluded that the following sequence of events occurred on the fateful morning of the "killer robot" tragedy:
While the software module written by Randy Samuels did cause the Robbie CX30 robot to oscillate out of control and attack its human operator, a good interface design would have allowed the operator to terminate the erratic robot behavior. Based upon an analysis of the robot user interface using Shneiderman's eight golden rules, this interface design expert has come to the conclusion that the interface designer, and not the programmer, was the more guilty party in this unfortunate fiasco.
The Case of the Killer Robot is a fictional scenario for ethics teaching and discussion purposes.
The site is updating and expanding and is currently in beta form. We encourage you to browse and see all that the Online Ethics Center has to offer. We are actively working to improve your experience on the site and hope you find what you are looking for.
We would be delighted to hear your thoughts on the site reorganization, and if you need help finding a specific resource, please contact us via email at firstname.lastname@example.org.