Testing large-scale computing systems is complicated, time-consuming, and therefore, very expensive. Comprehensive and thorough testing at the system level may entail tests directed to functions and performance of processors, cache coherency, memory, and input/output (I/O). Companies will sometimes have one or more groups of people dedicated to testing computing systems.
It is not uncommon for a test group to have accumulated a large number of tests over years of system testing. The tests represent years of accumulated effort and expertise. Since the tests themselves have been debugged, run against various different computing platforms, and at some time proved useful in verifying the integrity of a system or uncovering defects, reuse of tests may reduce the overall test effort.
Tests are generally implemented as a set of one or more executable programs, and each test is constructed to target some aspect of the system. A test generally requires the user to specify some amount of information that describes the system configuration. For example, the tester (or “end user”) may be required to specify identifiers for processors as well as the physical and/or logical disks in the system. The end-user knows, for example, that the system has eight processors—but may be unaware that these processors are distributed across two nodes with both local and shared caches. The end-user may be aware that the target system has C and D logical disk drives, but the user may not know that these drives are located on two different physical disks, use different device drivers, and reside on two different I/O buses.
If the end-user requires that a certain controller such as a specific device driver or a particular bus be the target of a test, the user is responsible for translating the requirement into terms the test will understand. This may require significant effort and investigative work, which may need to be repeated each time the test is moved to another system, or the configuration is changed on a system on which a test is to be rerun. Changes in hardware may make diagnosing hardware problems difficult and time-consuming.
Incorporating the capability to recognize particular configuration characteristics into individual tests may not be feasible. For example, tests may come from a variety of sources and may differ considerably in parameterization and input methods. In addition, tests may be written in different languages and have different internal structures. Some tests may be well documented, and others may be undocumented. For some tests, the source code may not be available.
Even if individual tests could be modified, enhanced, or adapted to recognize internal system hardware infrastructure, the effort and maintenance may outweigh the benefits. The tester would still be required to manually input parameters, schedule and monitor tests, and determine the appropriate mix of tests necessary to ensure adequate testing of the entire hardware system
A method and system that address these and other related issues are therefore desirable.