An Empirical Evaluation of Regular and Extreme Mutation Testing for Teaching Software Testing

Abstract

Teaching software testing can be challenging due to low student interest, high cognitive load, and lack of alignment with industry needs. Previous research has attempted to address these challenges by using mutation testing, which involves intentionally introducing faults into the code to measure the ability of a test suite to detect faults. Although this method has been proven effective in teaching software testing, it can sometimes be difficult for a novice to write a test to kill some mutants because they are too subtle and there are no hints. In contrast, extreme mutation testing involves more evident changes (e.g., removing a method body) that may be easier for novice testers to identify. This paper investigates extreme mutation testing as an alternative to teaching software testing by comparing it to regular mutation testing in an empirical evaluation with two undergraduate classes. Our results show that both can help teach software testing, with regular mutation testing slightly more effective, and both types of reports were considered clear by a similar number of students.

Martin Balfroid

PhD Student

Pierre Luycx

Msc Student

Benoît Vanderose

Assistant Professor of Software Engineering

Xavier Devroey

Assistant Professor of Software Testing

My research goal is to to ease software testing by exploring new paths to achieve a high level of automation for test case design, generation, selection, and prioritization. My main research interests include search-based and model-based software testing, test suite augmentation, DevOps, and variability-intensive systems.