Comparing the Effectiveness of Testing Methods in Improving Programs: the Effect of Variations in Program Quality

By Lorenzo Strigini and Michele Pizza; Proc. of the Ninth International Symposium on Software Reliability Engineering, ISSRE '98, Paderborn, Germany, November 1998.

ABSTRACT
We compare the efficacy of different testing methods for improving the reliability of software. Specifically, we use modelling to compare "operational" testing, in which test cases are chosen according to their probability of occurring in actual use of the software, against "debug" testing methods, in which the testers look for test cases which they consider likely to cause failure, or that satisfy some coverage criterion. We base our comparisons on the reliability reached by the program at the end of testing. Differently from previous studies, we consider the probability distribution of the achieved reliability, and thus the probability of satisfying specific requirements, rather than just the average reliability achieved. We take account of two sources of variation: the variation between the actual test histories that are possible for a given program and a given test method; and the fact that different programs start testing with different faults and initial reliability levels. By necessity, we use very simplified models of reality. Yet, we can show some interesting conclusions with important practical consequences. In general, there are stronger arguments in favor of operational testing than previous studies have shown.

The full text of this paper is available in .pdf and .ps format.

Figure 2 Correction

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

CSR Home | CSR Research Projects | CSR Publications | School of Informatics | City University