Evaluation & Assessment

In 1980, educators gathered at the Hastings Center to discuss whether there were any common goals for teaching ethics in higher education. Consensus was reached that the goals of teaching ethics, regardless of professional discipline are to assist the students in:

  1. Stimulating their moral imagination
  2. Recognizing ethical issues
  3. Analyzing key concepts and principles
  4. Eliciting their sense of responsibility
  5. Dealing constructively with ambiguity and disagreement

(From Callahan, Daniel. "Goals of Teaching Ethics" in Callahan, Daniel and Sissela Bok, Ethics Teaching in Higher Education. New York: Plenum Press, 1980. pp.64-69.)

Developing an effective evaluation strategy to measure student’s growth in these areas can be a challenge. What follows are some examples of assessment methods which have been used to measure the effectiveness of individual ethics instruction methods and institution-wide programs. Each is followed by a bibliography that describes these assessment methods in more detail.

Graded Assignments (can be worked into a course or workshop):

Homework Problems – Students can be asked to read a short case study outside of class, either as a separate assignment or as part of a problem set, and be asked to respond. Assignments should usually include some general guidance on how to respond to the case, such as the seven step method, or a series of more specific questions. (1)

Exam Questions – Much like an in-class version of the homework cases, students are given a short case to read and can be asked to either evaluate the responses of the participants, or to identify the ethical issues the case and formulate and defend a course of action. (2)

Essays – Students can be asked to write a longer response to a more complex case. This option has the advantage of helping students understand most of the issues in one case better than any one of the issues in shorter homework cases. (3)

A “one-minute” essay – students write a few sentences at the end of a discussion summarizing its content and what they picked up on. This allows the instructor to see what the student caught and helps the student organize and retain the experience before it begins to fade.

Lab assignment - Give an ethics case as a group lab assignment. Ask the students to discuss, and prepare a “lab report” of their discussion.

Role-playing – Students can either be assigned a role in a situation a few days before and try to act as they think one should in that situation.

Have students do an ethical analysis of their class projects using something similar to the following checklist.

  1. Identify stakeholders and their interests
  2. Identify the standards or norms they are using to make decisions about
    1. Technological developments
    2. Economic impact
    3. Safety
    4. Public health
    5. Environmental impacts
  3. Assess whether they are adhering to the professional guidelines, such as the NSPE Code of Ethics in an engineering course.
  4. Review their project from at least three ethical perspectives. (4)

For more ideas, see Davis, Michael. Ethics and the University. New York: Routledge, 1999. pp. 143-174.

Grading Ethics Assignments:

Often, instructors are wary of grading ethics assignments. They may feel that any grades they give out will be subjective. Because morals are in some way a product of feelings, religious upbringing, and culture, does an instructor have a right to give out these types of grades? This can be answered by asking yourself, “What did I teach my students about professional ethics?” and the follow-up question, “Can I grade them on that?” (4). For example, if one of your main goals was to make students sensitive to the ethical issues of safety, did the student manage to identify the relevant ethical issues in the case or problem? Fair grading will measure a student’s ability to recognize an ethical issue and analyze possible courses of action, not a student’s personal values or moral beliefs.

One tool for grading longer, more complex essays and case study responses is the Pittsburgh-Mines Engineering Ethics Assessment Rubric. This grading matrix was developed by a team of researchers in engineering, philosophy, and bioethics from the University of Pittsburgh and the Colorado School of Mines. (6) While made for analyzing students’ analyses of engineering ethics cases, variations of this kind of rubric could be used for almost any assignment of this kind. The rubric measures the following five attributes on a scale of 1 (lowest) through 5 (highest).

  • Recognition of dilemmas – (1) students fail to see problem; (5) students clearly identify key ethical issues.
  • Information – (1) students ignore important facts; (5) students identify unknown facts and use their own expertise to add appropriate information
  • Analysis – (1) students provide no analysis; (5) students cite analogous cases, offer more than one alternative solution, and identify risks for each solution.
  • Perspective – (1) students have wondering perspective; (2) students have one perspective; (5) students have global perspective. (6)
  • Resolution – (1) No resolution, resolution lacks integrity; (5) resolves case thoroughly through clear argumentation and understands consequences of various actions.

The Intercollegiate Ethics Bowl is a competitive tournament where teams of undergraduate students are asked questions about a case study and a panel of judges is asked to evaluate their answers based on the quality of their arguments and how they apply critical thinking to difficult situations. The judging criteria could easily be adapted to be used for individuals or small groups. The following criteria are used.

  1. Was the presentation clear and systematic? Regardless of whether or not you agree with the conclusion, did the team give a coherent argument in a clear and succinct manner? (10 points)
    • 1-2 = Incoherent presentation
    • 3-4 = Serious logical problems in the argument (poor)
    • 5-6 = Hard to follow the argument (passable)
    • 7-8 = Reasonably clear and systematic
    • 9 = Crystal clear presentation
    • 10= Exceptional
  2. Did the team avoid ethically irrelevant issues? Or was the team preoccupied with issues that are not ethically relevant or are of minor ethical relevance to the case?
    • 1-2 = Whole argument irrelevant
    • 3-4 = Major irrelevance in the argument (poor)
    • 5-6 = Some distractions from main argument (passable)
    • 7-8 = Minor irrelevancies
    • 9 = Exactly on point
    • 10 = Exceptional
  3. Did the team’s presentation clearly identify and thoroughly discuss the central moral dimensions of the case?
    • 1-2 = Failure to cover any relevant moral dimensions
    • 3-4 = Serious missing or underdeveloped dimensions (poor)
    • 5-6 = Some significant dimensions are missing or poorly covered (passable)
    • 7-8 = Most dimensions are present and well developed
    • 9 = All dimensions present and clarified appropriately
    • 10 = Exceptional
  4. Did the team’s presentation indicate both awareness and thoughtful consideration of different viewpoints, including especially those that would loom large in the reasoning of individuals who disagree with team’s position?
    • 1-2 = Minimal awareness of different viewpoints
    • 3-4 = Minimal consideration of different viewpoints…. (poor)
    • 5-6 = Underdeveloped discussion of different viewpoints…. (passable)
    • 7-8 = Solid analysis and discussion of different viewpoints, including careful attention especially to those that would loom large….
    • 9 = Insightful analysis and discussion of different viewpoints, including full and careful attention especially to those that would loom large….
    • 10 = Exceptional

Assessment of Course’s Effectiveness

Surveys are one informal way to get feedback on the satisfaction level of students, and can help improve ethics instruction from year to year. From 1993-2003, Professor Michael Davis at the Illinois Institution of Technology asked past participants of an Ethics Across the Curriculum workshop to have their students fill out a survey evaluating the effectiveness of ethics teaching in their course. The questions were as follows: 

  • Did this course do anything to change your awareness of ethics issues likely to arise in your profession or job?   (89% answered “yes”)
  • Did this course do anything to change your understanding of the importance of professional or business ethics?   (Just over 74% answered “yes”)
  • Did this course do anything to improve your ability to deal with the ethical issues it raised?   (Almost 78% answered “yes”)
  • In your opinion, did this course spend too much time on professional business ethics, too little, or just the right amount.   (Almost 70% answered “right amount, 10% answered “no.”) (7)

 For a full analysis of the survey results see “A Symposium – Integrating Ethics in Engineering and Science Courses” in the October 2005 edition of Science and Engineering Ethics

A pre and post-test can be used to measure student’s improvement over the course of the semester without taking up much class time.    For example, students can be asked to respond to a series of questions about a case study at the beginning of the ethics program, and the same questions can be asked about a similar case study included in the final exam. The answers from these two tests can then be compared.

Portfolios can be used to assess students’ development over a longer period of time, just as they traditionally are in education and creative arts programs. Students can be asked to complete a number of assignments over a period of time such as writing essays, responding to case studies, collecting stories from the popular press dealing with ethics, and so on, which they submit for grading at the end of the semester. (4)

(1)    Davis, Michael. Ethics and the University. New York: Routledge, 1999. pp.169
(2)    Davis, 170
(3)    Davis, 170.
(4)    Steneck, Nicholas H. “Designing Teaching and Assessment Tools for an Integrated Engineering Ethics Curriculum.” Presentation at the 29th ASEE/IEEE Frontiers in Education Conference. November 10-13, 1999 San Juan, Puerto Rico.

 

More Assessment Resources 

Bebeau, Muriel J. “The Defining Issues Test and Four Component Model: Contributions to Professional Education.” Journal of Moral Education. 31:3 (September 2002) 271-295.
Describes the development of a standardized test that can be used to measure the growth of moral reasoning skills in students over time.

Clancy, Edward A, Paula Quinn, and Judith E. Miller. “Assessment of a Case Study Laboratory to Increase Awareness of Ethical Issues in Engineering.” IEEE Transactions on Education. 48:2 (May 2005) 313-17.
This article discusses the assessment of a three-hour “laboratory period,” during which students read and discussed three short cases on engineering ethics.  The assessment included focus groups and surveys, and while in focus groups students agreed that this activity enhanced their awareness of ethical issues, the survey results, however, were equivocal. 

Kirkman, Robert. “Teaching for Moral Imagination: Assessment of a Course in Environmental Ethics.” Teaching Philosophy 31:4 (2008) 333-50.
This article looks at the results of an assessment project on a course in environmental ethics whose goals were to measure the impact of the course on students, as well as to contribute to a broader goal of developing assessment tools for ethics education.

Rudnicka, Ewa A. “Ethics in an Operations Management Course.” Science and Engineering Ethics. 11:4 (October 2005) 645-654.
Article includes a model of a grading rubric for evaluating students' understanding of ethics case studies.

Steneck, Nicholas H. "Designing Teaching and Assessment Tools for an Integrated Engineering Ethics Curriculum." Proceedings of the 29th ASEE/IEEE Frontiers in Education Conference. (1999): 12d6-11, 12d6-17. 
Describes how the faculty at the College of Engineering at the University implemented an across-the-curriculum approach for teaching engineering ethics, the development of strategic goals that shaped the program’s design, and the development of numerous assessment techniques to measure the effectiveness of the program. 

Barry, B.E. and Ohland, M.W. (2009), "Applied Ethics in the Engineering, Health, Business, and Law Professions: A Comparison." Journal of Engineering Education. 98(4) 377-388.
Based on the debate associated with the instruction and assessment of applied engineering ethics, this study explored what could be learned by looking across professions. The authors looked at literature at the intersection of ethics terminology, historical development, instruction and assessment within engineering, health, business and law.

 

See Also: Evaluation Tools in the OEC Resources section.

Cite this page: "Evaluation & Assessment" Online Ethics Center for Engineering 3/26/2010 National Academy of Engineering Accessed: Saturday, December 20, 2014 <www.onlineethics.org/Resources/Bibliographies/evaluationbiblio.aspx>