  For software interacting directly with real-world end-users, it is common
practice to script scenario tests validating the system's compliance with a
number of its features. However, these do not accommodate the replication of
the type of end-user activity to which the system is required to respond in a
live instance. It is especially true as compliance might also break in
scenarios of interactions with external events or processes, such as other
users. State-of-the-art approaches aim at inducing the software into runtime
errors by generating tests that maximize some target metrics, such as code
coverage. As a result, they suffer from targeting an infinitely large search
space, are severely limited in recognizing error states that do not result in
runtime errors, and the test cases they generate are often challenging to
interpret. Other forms of testing, such as Record-Replay, instead fail to
capture the end-users' decision-making process, hence producing largely
scripted test scenarios. Therefore, it is impossible to test a software's
compliance with unknown but otherwise plausible states. This paper introduces
"Synthetic End-User Testing," a novel testing strategy for complex systems in
which real-world users are synthesized into reusable agents and employed to
test and validate the software in a simulation environment. Hence, it discusses
how end-user behavioral examples can be obtained and used to create agents that
operate the target software in a reduced search space of likely action
sequences. The notion of action expectation, which allows agents to assert the
learned compliance of the system, is also introduced. Finally, a prototype
asserting the feasibility of such a strategy is presented.

Software Engineering