Richard G. Epstein, Westchester University of Pennsylvania
Mike Melamed, CWRU 2000
The Case of the Killer Robot is a detailed scenario that combines elements of software engineering and computer ethics.
The scenario consists of fictitious articles that touch on specific issues in software engineering and computer ethics. The articles discuss programs such as programmer psychology, team dynamics, user interfaces, software process models, software testing, the nature of requirements, software theft, and privacy. A major consideration is "when is the software good enough?"
The articles in the scenario begin with the indictment for manslaughter of a programmer who wrote faulty code that caused the death of a robot operator. Slowly, over the course of many articles, students are introduced to factors within the software company that also contributed to the accident. They are shown software development as a social process. It is hoped that students will begin to realize the complexity of the task of building real-world software and to see some of the ethical issues intertwined in that complexity.
This scenario is about 70 pages long and includes some tongue-in-cheek humor.
For permission see author information.
Special to the Silicon Valley Sentinel-Observer
Silicon Valley, USA
Cindy Yardley, a software tester at Silicon Techtronics, admitted today that she was the person who created the fraudulent "killer robot" software tests. The fraudulent tests were revealed earlier this week by Silicon Valley University professor Wesley Silber in what has come to be known as the "Silber Report".
At issue are quality assurance procedures that were performed on the program code written by Randy Samuels, the programmer charged with manslaughter in the killer robot incident. The Silber Report asserted that the test results reflected in internal Silicon Techtronics documents were inconsistent with the test results obtained when the actual killer robot code was tested.
In a startling development at noontime yesterday, Max Worthington, Chief Security Officer for Silicon Techtronics, announced his resignation at a packed news conference that was broadcast live by CNN and other news organizations.
Worthington stunned the assembled reporters when he began his news conference with the announcement, "I am Martha," referring to the fictional name this paper has used to identify an anonymous source inside Silicon Techtronics.
Worthington described his responsibilities at Silicon Techtronics in this way: "Basically, my job was to protect Silicon Techtronics from all enemies, domestic and foreign. By foreign I mean adversaries from outside the corporation. My role was mostly managerial. Those working under me had many responsibilities, including protecting the physical plant, watching out for industrial spying and even sabotage. I was also responsible for keeping an eye out for employees who might be abusing drugs or who might be disloyal in some way to Silicon Techtronics."
Worthington then pointed to a stack of bound volumes which were on a table to his left. "These volumes represent just some of the electronic surveillance of employees that I conducted over the years for my superior, Mr. Waterson. These are print outs of electronic mail messages that Silicon Techtronics employees sent to one another and to persons at other sites. I can say with great certainly that no employee was ever told that this kind of electronic surveillance was being conducted. However, I think the evidence shows that some employees suspected that this might be going on."
Several reporters shouted questions asking who at Silicon Techtronics knew about the electronic surveillance.
Worthington replied, "No one knew about this except Mr. Waterson, myself, and one of my assistants, who was responsible for conducting the actual monitoring. My assistant produced a special report, summarizing e-mail activity once a week, and that report was for Waterson's eyes and my eyes, only. Upon request, my assistant could produce a more detailed accounting of electronic communications."
Worthington explained that he was making the e-mail transcripts available to the press because he wanted the whole truth to come out concerning Silicon Techtronics and the killer robot incident.
The e-mail messages between employees at Silicon Techtronics indeed revealed new facets of the case. A message from Cindy Yardley to Robotics Division Chief Ray Johnson indicates that she faked the test results at his request. Here is the text of that message:
Re: samuels software
I have finished creating the software test results for that troublesome robot software, as per your idea of using a simulation rather than the actual software. Attached you will find the modified test document, showing the successful simulation.
Should we tell Randy about this?
Johnson's response to Yardley's message suggests that he suspected that electronic mail might not be secure:
Re: samuel's software
I knew I could count on you! I am sure that your devotion to Silicon Techtronics will be repaid in full.
Please use a more secure form of communication in the future when discussing this matter. I assure you that the way we handled this was completely above board, but I have my enemies here at good ol' SiliTech.
These communications were exchanged just a few days before the Robbie CX30 robot was shipped out to Cybernetics, Inc. This fact is important because the fake software tests were not part of a cover-up of the killer robot incident. These facts seem to indicate that the purpose of the fake software tests was to make sure that the Robbie CX30 robot was delivered to Cybernetics by the deadline that was negotiated between Silicon Techtronics and Cybernetics.
The e-mail transcripts reveal repeated messages from Ray Johnson to various people to the effect that the Robotics Division would definitely be closed down if the Robbie CX30 project was not completed on time. In one message, he lectures project leader, Sam Reynolds, on the "Ivory Snow Theory":
Re: don't be a perfectionist!
You and I have had our differences, but I must tell you that I like you personally. Please understand that everything I am doing is for the purpose of SAVING YOUR JOB AND THE JOB OF EVERYONE IN THIS DIVISION. I view you and all of the people who work with me in the Robotics Division as my family.
Waterson has made it clear: he wants the robot project completed on time. That's the bottom line. Thus, we have no recourse but "Ivory Snow". You know what I mean by that. It doesn't have to be perfect. The user interface is our fall back if this version of the robot software has some flaws. The robot operator will be safe because the operator will be able to abort any robot motion at any time.
I agree with you that the non-functional requirements are too vague in places. Ideally, if this weren't crunch time, it would be good to quantify the amount of time it would take the operator to stop the robot in case of an accident. However, we cannot renegotiate those now. Nor, do we have time to design new tests for new, more precise nonfunctional requirements.
I cannot emphasize enough that this is crunch time. It's no sweat off Waterson's back if he lops off the entire Robotics Division. His Wall Street friends will just say, "Congratulations!" You see, to Waterson, we are not a family, we are just corporate fat.
In this message, Ray Johnson seems to be less concerned with the security of communicating by e-mail.
Neither Ray Johnson nor Sam Reynolds could be reached for comment. However, the Sentinel-Observer interviewed Cindy Yardley at her home yesterday evening.
Yardley was obviously upset that her private e-mail messages had been released to the press. "I am relieved in some ways. I felt tremendous guilt when that guy was killed by a robot that I helped to produce. Tremendous guilt."
The Silicon-Observer asked Ms. Yardley whether she felt that she had made an ethical choice in agreeing to fake the software test results. She responded with great emotion: "Nothing, nothing in my experience or background prepared me for something like this. I studied computer science at a major university and they taught me about software testing, but they never told me that someone with power over me might ask me to produce a fake software test!"
"When Johnson asked me to do this, he called me to his office, as if to show me the trappings of power. You see, someday I would like to be in a managerial position. I sat down in his office and he came right out and said, 'I want you to fake the test results on that Samuels software. I don't want Reynolds to know anything about this.'"
Yardley fought back tears. "He assured me that no one would probably ever see the test results because the robot was perfectly safe. It was just an internal matter, a matter of cleanliness, in case anyone at Cybernetics or higher up in the corporation got curious about our test results. I asked him whether he was sure about the robot being safe and all that and he said, 'It's safe! The user interface is our line of defense. In about six months we can issue a second version of the robotics software and by then this Samuels problem will be solved.'"
Yardley leaned forward in her chair as if her next remark needed special emphasis. "He then told me that if I did not fake the software tests, then everyone in the Robotics Division would lose their job. On that basis I decided to fake the test results. I was trying to protect my job and the jobs of my co-workers."
Yardley is currently pursuing an MBA degree at night at Silicon Valley University.
The Sentinel-Observer then asked Ms. Yardley whether she still felt that she had made an ethical decision, in view of the death of Bart Matthews. "I think I was misled by Ray Johnson. He told me that the robot was safe."
Another revelation, contained in the released e-mail transcripts, was the fact that Randy Samuels stole some of the software that he used in the killer robot project. This fact was revealed in a message Samuels sent to Yardley when she first tested his software and got erroneous results:
Re: damned if I know
I cannot for the life of me figure out what is wrong with this function, swing_arm(). I've checked the robot dynamics formula over and over again, and it seems to be implemented correctly. As you know, swing_arm() calls 14 different functions. I lifted five of those from the PACKSTAT 1-2-3 statistical package verbatim. Please don't tell a soul! Those couldn't be the problem, could they?
Experts tell the Silicon-Observer that lifting software from a commercial software package such as the immensely popular PACKSTAT 1-2-3 is a violation of the law, because software is protected by the same kind of copyright that protects printed materials.
Mike Waterson, CEO of Silicon Techtronics issued an angry statement concerning Max Worthington's release of "confidential" e-mail transcripts. Waterson's statement said, in part, "I have asked our attorneys to look into this matter. We consider those transcripts the exclusive property of Silicon Techtronics. Our intent is to pursue either civil or criminal charges against Mr. Worthington."
In reaction to yesterday's developments in the killer robot case, Association for Computing Machinery (ACM) announced its intention to investigate whether any ACM members at Silicon Techtronics have violated the ACM code of ethics. The ACM is an international association of computer scientists with 85,000 members.
ACM president Dr. Turina Babbage issued a statement from the ACM's Computer Science Conference, which is held every winter, this winter in Duluth, Minnesota. She said:
"All members of the ACM are bound by the ACM Code of Ethics and Professional Conduct. This code states, in part, that ACM members have the general moral imperative to contribute to society and human well-being, to avoid harm to others, to be honest and trustworthy, to give proper credit for intellectual property, to access computing and communication resources only when authorized to do so, to respect the privacy of others, and to honor confidentiality.1
Beyond that, there are professional responsibilities, such as the obligation to honor contracts, agreements, and assigned responsibilities, and to give comprehensive and thorough evaluations of computing systems and their impacts, with special emphasis on possible risks.
Several of the people involved in the killer robot case are ACM members and there is cause to believe that they have acted in violation of our association's code of ethics. Therefore, I am asking the ACM Board to appoint a task force to investigate ACM members who might be in gross violation of the code.
We do not take this step lightly. This sanction has been applied only rarely, but the killer robot incident has not only cost a human life, but it has done much to damage the reputation of the computing profession.
The site is updating and expanding and is currently in beta form. We encourage you to browse and see all that the Online Ethics Center has to offer. We are actively working to improve your experience on the site and hope you find what you are looking for.
We would be delighted to hear your thoughts on the site reorganization, and if you need help finding a specific resource, please contact us via email at firstname.lastname@example.org.