The present disclosure relates to testing in general, and to generation of a system-level testcase, in particular.
Computerized devices control almost every aspect of our life—from writing documents to controlling traffic lights. However, computerized devices are bug-prone, and thus require a testing phase in which the bugs should be discovered. The testing phase is considered one of the most difficult tasks in designing a computerized device. The cost of not discovering a bug may be enormous, as the consequences of the bug may be disastrous. For example, a bug may cause the injury of a person relying on the designated behavior of the computerized device. Additionally, a bug in hardware or firmware may be expensive to fix, as patching it requires call-back of the computerized device. Hence, many developers of computerized devices invest a substantial portion, such as 70%, of the development cycle to discover erroneous behaviors of the computerized device.
During the testing phase, functional verification may be performed. One approach to functional verification is to perform simulation of a system-under-test (SUT). The simulation operates based on stimuli that may be generated. In some exemplary embodiments, a testcase generator is operative to generate a testcase to provide for stimuli of the SUT. The testcase may be generated based upon a test template. In some exemplary embodiments, the testcase may be a list of instructions to the SUT which are in turn simulated by the simulator. The test template may be a descriptive specification of a set of testcases. The test template may be provided using, for example, a context-free formal language. The test template may comprise commands as to how the generation of the testcase is to be performed. These commands generally define generation tasks, such as for example setting an address to be used in the instruction, selecting an instruction out of a set of instructions, determining a number of repetitions to be performed, determining whether a synchronization event occurs, or the like. Generally speaking, based on the test template a plurality of alternative testcases may be generated, all are characterized in fulfilling the requirements of the test template. In some cases, the test template may explicitly or implicitly define generation tasks. For example, in some cases, the test template may describe a template of a resulting testcase, and therefore implicitly define generation tasks.
An offline generator is designed to generate a testcase, which is in turn utilized by the simulator for simulation. An online generator is designed to operate in generation-simulation cycles. In each cycle one or more instructions of the testcase may be generated, and in turn simulated by the simulator. Based on the simulated execution of the previous cycle, generation of consecutive instruction(s) may be performed.
In some exemplary embodiments, a generator may be a general-purpose generator, such that may employ a Constraint Satisfaction Problem (CSP) solver to generate the testcase based on the test template. The CSP solver may, for example, be utilized to determine an instruction that is in line with the requirements of the test template and that is optionally biased towards a quality testcase. In some exemplary embodiments, biasing may be architecture-aware, such as to invoke specific events of the specific architecture of the SUT (e.g., cache events, page faults, segmentation faults, resource contentions, address collisions or the like). In some exemplary embodiments, the test template is translated into a series of CSPs comprising of hard constraints and soft constraint, wherein the soft constraints are utilized to bias the testcase towards a higher quality testcase, when possible.
In some exemplary embodiments, the generator may be a special-purpose generator, such as specifically designed for the purpose of generating testcases for the SUT. A dedicated generator may be utilized when a CSP-based approach is not efficient. For example, a CSP approach of a cryptographic engine which encodes and decodes inputs may require the CSP solver to solve a relatively hard computational problem of computing the encryption function. Use of a dedicated solver may provide for a more efficient solution.