Quickly Generating Diverse Valid Test Inputs with Reinforcement Learning
Property based testing is a popular approach for validating the logic of a program. An effective property based test quickly generates many diverse valid test inputs and runs them through a parameterized test driver. However, when the test driver requires strict validity constraints on the inputs, completely random input generation fails to generate enough valid inputs. Existing approaches to solving this problem rely on whitebox or greybox information collected by instrumenting the input generator and/or test driver. However, collecting such information reduces the speed at which tests can be executed. In this paper, we propose and study a black box approach for generating valid test inputs. We first formalize the problem of guiding random input generators towards producing a diverse set of valid inputs. This formalization highlights the role of a guide which governs the space of choices within a random input generator. We then propose a solution based on reinforcement learning (RL) using a tabular, on policy RL approach to guide the generator. We evaluate this approach, RLCheck, against pure random input generation as well as a state of the art greybox evolutionary algorithm, on four real world benchmarks. We find that in the same time budget, RLCheck generates an order of magnitude more diverse valid inputs than the baselines.
2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)