Regression test selection for C++ based on call graph analysis

Abstract

Regression testing -- running available tests after each project change -- is widely practiced in industry to check that a project change does not break working functionalities. Despite its widespread use and importance, regression testing is a costly activity, and the cost is steadily increasing with the increase in the number of tests and the number of changes. Regression test selection (RTS) attempts to optimize regression testing activity by deselecting tests not affected by project changes and executing remaining tests. RTS has been extensively studied and several tools have been deployed in industry. However, work on RTS over the last decade has mostly focused on managed languages (e.g., Java). Meanwhile development practices (e.g., frequency of commits, testing framework, compilers, etc.) in C++ projects have dramatically changed, and the way we should design and implement RTS tools and the benefits of those tools is unknown. This thesis presents a design and implementation of an RTS technique, named Ekstazi++, that targets projects written in C++, which use the LLVM compiler and the Google Test testing framework. Ekstazi++ implements an RTS technique based on the call graph analysis. Ekstazi++ integrates with many existing build systems, including AutoMake, CMake, and Make. We evaluated Ekstazi++ on 11 large open-source projects, totaling 3,811,916 lines of code. We measured the benefits of Ekstazi++ compared to running all available tests (i.e., retest-all) in terms of the number of executed tests, as well as end-to-end testing time. Our results show that Ekstazi++ reduces the number of executed tests and end-to-end testing time up to 97.20% and 88.09%, respectively.