Author(s): Michael McFarland, S.J.
Originally published by the Markkula Center for Applied Ethics
Wayne Davidson is a software engineer in the aerospace division of Occidental Engineering, a large engineering firm. For the past two years he has been working as a test engineer for Operation Safe Skies, a project to build a prototype of the next generation air traffic control system. This project, which is funded by a contract from the Federal Aviation Agency (FAA), is a very important one for Occidental. With all the cutbacks in defense spending, the...
Wayne Davidson is a software engineer in the aerospace division of Occidental Engineering, a large engineering firm. For the past two years he has been working as a test engineer for Operation Safe Skies, a project to build a prototype of the next generation air traffic control system. This project, which is funded by a contract from the Federal Aviation Agency (FAA), is a very important one for Occidental. With all the cutbacks in defense spending, the aerospace division has been losing business. The Safe Skies project has provided much needed business, and could lead to a much larger contract if successful. Mindful of its strategic importance, the company had bid very aggressively for the original contract. In fact they had "low-balled" it, bidding less than it would take to do the work properly. They felt that was the only way they could beat out their competitors, who were just as hungry for the work. Because of their somewhat shaky financial position, the company was not willing to take a loss on the project, so the project has been underfunded and understaffed. Nevertheless those working on the project have made a heroic effort, working eighteen hour days seven days a week to meet the deadline, because they know how much it means to the company, not to mention their own jobs. They are now very close to success.
A version of the prototype has been completed and turned over to Wayne for testing. He has run extensive simulations on it and found that it works as it should except for one little problem. When there are too many aircraft in the system, it will sometimes lose track of one or more of them. The "forgotten" aircraft will simply disappear from the screen, there will be no trace of it anywhere, and it will be ignored by all of the collision avoidance and other safety tests. Wayne has been working with the software designers to identify the cause of the problem, and they have traced it to a subtle error in memory allocation and reuse. They are confident that they can fix it, but it will take a month or more to do the redesign, coding and testing.
Wayne meets with his boss, Deborah Shepherd, the project manager, to discuss the implications. She tells him that what he is asking for is impossible. The contract requires that the company deliver a fully certified, working version of the software in three days for system integration and test. The government has developed a new, get-tough policy on missed deadlines and cost overruns, and Occidental is afraid that if they miss this deadline, the government will make an example of them. They would be subject to fines and the loss of the remainder of the prototype contract; and they might not be allowed to bid on the contract for the full system. This would have a devastating effect on the aerospace division, resulting in thousands of lost jobs.
They consider whether they can do a quick patch to the software before turning it over, but Wayne adamantly refuses to release any code that has not been tested thoroughly. There is always a chance that the patch would interact with some other part of the program to create a new bug.
"Then we'll have to deliver the software as is," Deborah says. "I can't jeopardize this project or the jobs of my people by missing that deadline."
"We can't do that!" exclaims Wayne. "That's like delivering a car with defective brakes."
"Don't worry," Deborah reassures him. "We have contacts in the FAA, so we know their testing plans. They will do a lot of simulations to make sure the software works with the hardware and has all the functionality in the specs. Then they will do live tests, but only at a small airport, with a backup system active at all times. There is no way they will overload the system in any of this. After that they will have some change requests. Even if they don't, we can give them an updated version of the program. We can slip the bug fix in there. They will never see the problem. Even if they do, we can claim it was a random occurrence that would not necessarily show up in our tests. The important thing is no one is in any danger."
"Maybe they won't find the bug, but I know it's there. I would be lying if I said the system passed all the necessary tests. I can't do that. Anyway, it would be illegal and unprofessional."
"You can certify that it is safe, because it is, the way they are going to use it."
And so he does. In the end Wayne signs off on the software. It is delivered to the FAA and makes it through all the preliminary tests, including live tests at a small airport in the Midwest. As a result of these tests, the FAA requests some changes in the user interface, and when Occidental delivers the new software it includes a robust solution to the problem of the disappearing aircraft. No one outside of Deborah's group ever learns of the problem. In fact Occidental's success with the prototype leads to major contracts for air traffic control software, giving much-needed business to the aerospace division. This saves hundreds of jobs, and allows the company to add hundreds more.
Wayne Davidson, however, takes early retirement once the prototype project is finished, in order to write a book on software testing. He feels that the book should have a chapter on ethics, but he can never bring himself to write it.
What do you think about Wayne's decision? Was it ethical?
Next: Tutorial on Ethical Decision Making
Michael McFarland, S.J., a computer scientist, is the former president of College of the Holy Cross and was a visiting scholar at the Markkula Ethics Center. June 2012
Author(s): Michael McFarland, S.J.
Originally published by the Markkula Center for Applied Ethics
In practice when we are faced with an ethical dilemma, we seldom think it through from first principles. We usually draw on existing experience and past judgements. Making a sound ethical judgement is difficult. It involves consideration and weighing of many factors, the sifting through of different positions, careful reasoning and argument, and much experience of the implications of different actions, what works and what does not. It is helpful to be able to apply the wisdom so dearly bought by applying the judgements to new situations. There are a number of ways in which we do that.
In ethics as in law, we often decide a case by finding a similar case that has already been decided and applying the same judgement. When Wayne in the Occidental case protests that delivering flawed safety-critical software is like selling a car with defective brakes, he is reasoning by analogy. We are well aware of issues of safety in the automobile industry and the importance of engineers maintaining the highest standards for critical systems. Most would agree that it is wrong for engineers knowingly to release a vehicle with brakes that could fail. In the same way, Wayne argues, it would be wrong for him knowingly to release safety-critical software that could fail. The cases are analogous according to his analysis because they both involve engineered systems whose failures could cost lives, and both are being released with known flaws.
The use of analogy and arguing by cases goes back at least as far as the ancient Greek philosophers. For example in Plato's Gorgias,Socrates wishes to discredit sophists because of the shallowness of their flattering oratory, so he sets up the following analogy: "what pastry making is to medicine, oratory is to justice."33In other words, just as pastry gives the body an immediate sensation of well-being, though it is in fact unhealthy, whereas medicine produces real health, so the sophists' oratory seeks only the appearance of good, while justice is the true good of the state. The implication is that because there is something misleading and unhealthy about pastry, in the same way there is something untrue and corrupting about sophistry.
The ultimate justification for using analogy in ethical judgement is the principle of universalizability noted above as one of the principles derivable from Kant's categorical imperative. According to this principle if Case A and Case B are ethically analogous, meaning that they are governed by the same ethical principles and do not differ in any ethically significant way, then whatever judgement is attached to A should also be attached to B. If we have already arrived at a judgement for A, we can use it for B as well. The hard part is deciding when cases are analogous, and especially what differences are significant enough ethically to invalidate the analogy. For example, one might counter Wayne's use of the defective brakes analogy by pointing out that, while brakes have physical defects that cause them to fail randomly, the air traffic control program has a logical defect that fails in a predictable manner, and that removing a program bug is a routine procedure that can be done with a regularly scheduled program update, whereas retrofitting brakes is a major operation that can only be done through an extraordinary recall. But are these differences enough to invalidate the judgement based on a fundamental similarity, that both involve knowingly releasing a safety-critical system with a potentially dangerous flaw? For that matter, are software bugs really that predictable and controllable?
The above example shows that analogies can be enlightening, but they can never simply be taken at face value. They must always be examined very carefully. Nevertheless, even though analogies seldom give easy answers to ethical problems, they are often worth pursuing because of the insight they can provide. Thinking through the similarities and differences with known cases highlights the ethical issues and the ethically relevant factors, and suggests applicable ethical principles. At the very least comparative case analysis is a good way of exploring new issues and cases. For example, to what extent do our categories and rules about ownership of tangible property apply to software? What are the relevant similarities and differences? Can software be "owned" in the same way that a car or house can? Is copying software "stealing" or "sharing, " two competing analogies with quite different ethical implications.
Law is one of the ways in which society orders its common life. It concretizes some of society's expectations for justice, honesty, fidelity, and respect for rights. Thus the law reflects, in some complex and imperfect way, society's ethical requirements.
While there is a strong relationship between law and ethics, they are not identical. It is a mistake to think that when something is legal, it is necessarily ethical, or worse yet, that "It's ok as long as you get away with it." The law, at its best, represents the minimum expectations necessary to maintain a civilized society. There are many areas of private life it does not seek to regulate, and other areas that, while they touch public life, are left alone because it would be too oppressive to try to control them. Even in the areas that it can and should be concerned with, the law is a less than perfect representation of ethics, both because of disagreements and misunderstandings about what is right and because powerful interests often block legislation that would serve the public good at their expense. Therefore there are many actions that are legal but still not ethical. It may not be illegal in some cases to make misleading claims about a product, but it is unethical. It may not be illegal to take an unpublished idea from a colleague and publish it as one's own, but it is certainly wrong. In most places it is still not against the law to force employees to spend all day at a keyboard working under conditions that are likely to lead to very painful injuries, but it is still a violation of the duty not to harm others. Those who take the law as the ultimate standard of what is right and wrong are in an arrested stage of moral development. 34
On the other hand, not everything illegal is also immoral. There some laws whose requirements are arbitrary; they have nothing to do with right and wrong. For example, there is nothing fundamentally wrong with driving on the left side of the road. Yet in many countries the law forbids it. Some such law is necessary to impose at least a minimum of order on the flow of traffic, but the choice of right over left as the preferred side is arbitrary; other countries do as well driving on the left.
Other laws are simply wrong. The laws that maintained slavery in the United States, for example, were unjust, and those who violated them by refusing to return escaped slaves were not necessarily acting unethically. An even more striking example is the whole system of laws meant to facilitate the extermination of the Jews in Nazi Germany. Those who broke them by hiding Jews or helping them escape were not immoral. In fact they may have been the only people acting justly in that society. It can never be taken for granted that the law is right. There is always a responsibility to examine and critique it, and, if it is found to be seriously deficient, to try to change it.
Nevertheless, there is generally a presumption in favor of obeying the law, once it is established. There are a number of reasons for this. First, laws, imperfect as they are, often represent an attempt to codify society's standards of ethical behavior. In many cases a law is the product of a serious debate over ethical issues and an attempt to balance and reconcile competing values and interests. Also laws develop over time, as people learn more about their effects and reflect on their meaning. Therefore laws embody a certain collective wisdom. This in itself is enough to command serious consideration, if not uncritical acceptance. Second, there are usually penalties for violation of the law. To break a law is to put oneself, and often one's community or institution, in danger of fines and imprisonment, embarrassment and scandal, and other losses. Someone would have to have very grave reasons for taking on this risk. Civil disobedience is sometimes an important weapon in the struggle for justice; but it is not to be taken lightly. Third, laws, even when they are arbitrary, are needed to keep order in society, and to violate them is to violate that order. To drive on the left is not wrong in itself, but to drive on the left when the law says to drive on the right is to disrupt traffic flow unnecessarily and to endanger oneself and others. Finally, the law, when it works properly, defines a certain set of constraints under which everyone operates. When individuals or groups act outside the law, it gives them an unfair advantage over those who observe it. For example part of the cost of an industrial process is usually the disposal of wastes. If a company can simply dump them into the air or water, it is really transferring some of the costs to public agencies that must clean up the waste and the public, who pay in terms of a greater threat to their health and enjoyment. Clean air and water standards are meant to avoid that. When a company breaks these laws, it is avoiding its fair share of the cost of maintaining a livable environment and gaining an unfair advantage over companies that keep them.
When Wayne in the Occidental case protests that releasing the flawed software would be illegal, presumably because it involved fraud and violated the terms of their contract with the government, he is raising a very serious objection, even if the law is overly restrictive, as Deborah seems to think. They would be exposing themselves and the company to the risk of very serious consequences, including fines, imprisonment, disgrace, loss of customers, and loss of future government contracts. Furthermore, because the laws exist in part to keep the bidding process fair and make sure the bidders deliver what they promise, it would be an injustice to violate the law, for the reasons listed earlier in the section on justice.
Laws, therefore, by their very existence, create ethical obligations to observe them. At the same time, these laws are not above ethical analysis and criticism. When we examine an ethical issue, therefore, we must both understand what the law requires and ask whether the law is right or if it needs to be changed.
Norms are ethical "rules of thumb." They are guidelines and rules that have been developed for analyzing and deciding cases in a particular area of ethics, such as medical ethics or political ethics. They are less fundamental, and therefore less general and authoritative, than the basic duties listed earlier. Their authority comes from their having been found useful in practice and from the fact that they somehow embody our judgements about what is right in a given situation. Therefore they are a way of formalizing, generalizing and communicating ethical wisdom and experience in a specific field.
A good example of a set of norms is the so-called "just war theory." This is a set of tests that are to be applied to determine whether a particular war can be justified. 35 There are two parts to it. The first governs the decision to enter into a war. This is tolerable only if the following conditions are met:
In the computer field, an example of norms is the "Ten Commandments of Computer Ethics" published by the Computer Ethics Institute:
These norms are not as specific or precise as the just war theory, nor do they have the weight of tradition or the test of time behind them. Nevertheless they do represent a consensus among many computer ethicists on what constitutes ethical computer use, and can provide some general guidance to users.
Norms are different from laws in that they are meant to guide judgement rather than to regulate behavior. Norms are not meant to be enforceable and do not have the authority of government behind them, although in some cases, norms can be incorporated into laws. In general, however, norms must stand on their own; their authority comes from their own inherent wisdom and the moral authority of the community that has formulated and accepted them.
An important task of ethics is to formulate or recognize norms for particular areas of concern. This is far more valuable than endorsing or condemning specific acts. A good set of norms gives a person guidance in understanding new situations, in knowing what questions to ask and what factors to consider, and in formulating an ethical response.
Many professions, including doctors, lawyers, and the military, have their own specific codes of ethics. These are needed because these groups are entrusted with special responsibilities in the community, so that a higher degree of care and dedication to the common good is expected of them, and because in their work they deal with special issues and problems that ordinary citizens do not usually face. For example, physicians have had codes of ethics going all the way back to the Hippocratic Oath that had its origins in ancient Greek society. Now they are covered the codes of their professional societies, such as the code of the American Medical Association or the code of the British Medical Association. There is also an International Code of Medical Ethics. These are all attempts to define the physician's duty to the patient, the profession, and the society. They include the duties to preserve life, to act always for the welfare of the patient, to respect confidentiality, to act honestly and justly, and so on. 36
Professional codes of ethics tend to come from within their professions; part of the definition of a profession is that it is self-governing. In some cases the codes are enforceable. Serious violations can lead to suspension or expulsion from the profession and the dishonor that comes with it. However the moral authority of a professional code comes from the fact that it represents a consensus of the traditions of the profession and the judgments of its most respected members on the duties specific to their particular calling.
Computer scientists and engineers represent a relatively young field compared to law, medicine and other well-established professions. Nevertheless they have a number of professional organizations that have developed codes of ethics. These include the codes of the Association for Computing Machinery (ACM), the International Federation for Information Processing (IFIP), and the Data Processing Management Association (DPMA). The Institute of Electrical and Electronics Engineers (IEEE), which includes many computer professionals in its ranks, also has a code of ethics; and the IEEE Computer Society (IEEE-CS) and the ACM have jointly developed and approved a Software Engineering Code of Ethics and Professional Practice.
Of these the ACM code and the joint ACM/IEEE-CS code for software engineering are the most comprehensive. The ACM Code37 , revised in 1992, contains a preamble and four parts. The first is a set of "general moral imperatives" that link the responsibilities particular to computer professionals to fundamental ethical principles such as the duties to be honest and fair, to contribute to the good of society and humanity and avoid harm, and to respect privacy, confidentiality and intellectual property rights. Section 2 has more specific responsibilities of individual computer professionals. These include the obligation to understand and take responsibility for the consequences of their work, to honor contracts, laws and other obligations, to help educate the public, and to avoid unauthorized access to computers and communications systems. Section 3 balances the individualist perspective of section 2 by giving the obligations of those who have roles of leadership in organizations. These include the responsibility to see that the organization serves the needs, welfare and dignity of workers, users of its products, and others affected by its work. Finally there is a short section on compliance. This is seen as mainly voluntary, but there is a possibility of the ACM taking action against a member for serious code violations. The code proper is accompanied by a set of so-called Guidelines that serve as a commentary on and explanation of the code.
The joint ACM/IEEE-CS Software Engineering Code of Ethics and Professional Conduct38 contains a number of principles to guide the judgment and conduct of software engineers, organized around the stakeholders to whom they owe responsibility and other professional concerns, including the public at large, the client and employer, the product, professional judgment, management, the profession, colleagues and personal development. The code makes it clear that protecting the public interest takes priority over all other concerns. Under each principle is a list of specific obligations that illustrate how that principle is to be put into practice. For example, under the first principle, which states that "software engineers shall act consistently with the public interest," one of the specific obligations (1.06) requires that software engineers shall "Be fair and avoid deception in all statements, particularly public ones, concerning software or related documents, methods and tools." This is particularly relevant to the Occidental case discussed here. So is 5.11 under Management, which says that anyone managing a software engineer shall "Not ask a software engineer to do anything inconsistent with this Code."
These codes are not meant to be a comprehensive definition of ethical behavior for computer professionals, although the ACM Code has been applied in a number of cases. 39 Nor do they give an answer to every ethical dilemma; that is neither possible nor desirable. There must be respect for the ethical and professional autonomy of the individual member, and awareness of the need for individual judgement in particular cases. As the joint ACM/IEEE-CS Code states in its Preamble:
Ethical tensions can best be addressed by thoughtful consideration of fundamental principles, rather than blind reliance on detailed regulations. These Principles should influence software engineers to consider broadly who is affected by their work; to examine if they and their colleagues are treating other human beings with due respect; to consider how the public, if reasonably well informed, would view their decisions; to analyze how the least empowered will be affected by their decisions; and to consider whether their acts would be judged worthy of the ideal professional working as a software engineer. In all these judgments concern for the health, safety and welfare of the public is primary; that is, the "Public Interest" is central to this Code.
What these codes do provide is a set of shared expectations and obligations that can help define the profession and its commitment to serve the public. That in itself is important.
Return to Occidental Engineering Case Study
Next "Responsibility: Part 7"