Search-Based Software Testing
Researchers and practitioners have developed automated test case generators for various languages and platforms, each with different strengths. Comparing these tools requires large-scale empirical evaluations, which demand significant setup and analysis. The JUnit Generation benchmarking infrastructure (JUGE) automates the production and comparison of Java unit tests for multiple purposes, aiming to streamline evaluation, support knowledge transfer, and standardize practices. This master’s thesis will use the existing JUGE infrastructure to conduct an extensive empirical comparison of unit test generators for Java, and provide the resulting dataset to the research community.
The field of automated test case generation has grown considerably in recent years to reduce software testing costs and find bugs. However, the techniques for automatically generating test cases for machine learning libraries still produce low-quality tests and papers on the subject tend to work in Java, whereas the machine learning community tends to work in Python. Some papers have attempted to explain the causes of these poor-quality tests and to make it possible to generate tests in Python automatically, but they are still fairly recent, and therefore, no study has yet attempted to improve these test cases in Python. In this thesis, we introduce 2 improvements for Pynguin, an automated test case generation tool for Python, to generate better test cases for machine learning libraries using structured input data and to manage better crashes from C-extension modules. Based on a set of 7 modules, we will show that our approach has made it possible to cover lines of code unreachable with the traditional approach and to generate error-revealing test cases. We expect our approach to serve as a starting point for integrating testers’ knowledge of input data of programs more easily into automated test case generation tools and creating tools to find more bugs that cause crashes.
