Eugenio Alberdi, PhD, Andrey Povyakalo, PhD, Lorenzo Strigini, MEng, Peter Ayton, PhD
[recipient of the 2005 Herbert M. Stauffer Award for the "Best Clinical Paper" from the Association of University Radiologists]
Rationale and Objectives. To investigate the effects of incorrect
computer output on the reliability of the decisions of
human users. This work followed an independent UK clinical trial that
evaluated the impact of computer-aided detection
(CAD) in breast screening. The aim was to use data from this trial
to feed into probabilistic models (similar to those used
in “reliability engineering”) which would detect and assess possible
ways of improving the human–CAD interaction. Some
analyses required extra data; therefore, two supplementary studies
were conducted. Study 1 was designed to elucidate the
effects of computer failure on human performance. Study 2 was conducted
to clarify unexpected findings from Study 1.
Materials and Methods. In Study 1, 20 film readers viewed 60
sets of mammograms (30 of which contained cancer) and
provided “recall/no recall” decisions for each case. Computer output
for each case was available to the participants. The
test set was designed to contain an unusually large proportion (50%)
of cancers for which CAD had generated incorrect
output. In Study 2, 19 different readers viewed the same set of cases
in similar conditions except that computer output
was not available.
Results. The average sensitivity of readers in Study 1 (with
CAD) was significantly lower than the average sensitivity of readers
in Study 2 (without CAD). The difference was most marked for cancers
for which CAD failed to provide correct prompting.
Conclusion. Possible automation bias effects in CAD use deserve
further study because they may degrade human decision-
making for some categories of cases under certain conditions. This
possibility should be taken into account in the
assessment and design of CAD tools.
Key Words. Computer-aided detection (CAD); mammography; evaluation; reliability.
Acad Radiol 2004; 11:909–918