Doubts About Published Results

Author(s): Adapted by Albert R. Meyers and Caroline Whitbeck from a scenario contributed by an MIT Computer Science graduate student (December 1993)

You are a computer science graduate student and for two years have been working on an operating system design in Professor Carr's group. Professor Carr has designed a set of novel heuristics for file-system cache maintenance. Carr published performance graphs describing a simulation of a prototype file input/output subsystem in a journal article and included the graphs in the proposal for the group's current grant. The graphs indicate that Carr's heuristic methods will significantly improve file-system cache performance.

You devised a modification to the file-system cache heuristics and asked Carr how to run the simulation code to test the modification. Carr replied that the simulation code had not been used in a long time and had been archived to tape. Carr said it was not worth the trouble of trying to remember the archived filenames, because the simulation code was very poor and written in a language that does not run in the group's current computing environment. He told you to write a new, up-to-date simulator.

As you worked on the new simulator, you asked Carr how to simulate several classes of events, but Carr claimed not to remember these details of the old simulator. When you have finished building a new simulator, your results are considerably worse than those reported in the performance graphs that Carr published.

You now suspect that Carr did not do a previous simulation but made up the numbers in the performance graphs. Some of your own presentations and papers have been based on Carr's performance data. What can/should you do? What, if any, ambiguities do you face? What risks are there in this situation to you or to others?