  Performance variability management is an active research area in
high-performance computing (HPC). We focus on input/output (I/O) variability.
To study the performance variability, computer scientists often use grid-based
designs (GBDs) to collect I/O variability data, and use mathematical
approximation methods to build a prediction model. Mathematical approximation
models could be biased particularly if extrapolations are needed. Space-filling
designs (SFDs) and surrogate models such as Gaussian process (GP) are popular
for data collection and building predictive models. The applicability of SFDs
and surrogates in the HPC variability needs investigation. We investigate their
applicability in the HPC setting in terms of design efficiency, prediction
accuracy, and scalability. We first customize the existing SFDs so that they
can be applied in the HPC setting. We conduct a comprehensive investigation of
design strategies and the prediction ability of approximation methods. We use
both synthetic data simulated from three test functions and the real data from
the HPC setting. We then compare different methods in terms of design
efficiency, prediction accuracy, and scalability. In synthetic and real data
analysis, GP with SFDs outperforms in most scenarios. With respect to
approximation models, GP is recommended if the data are collected by SFDs. If
data are collected using GBDs, both GP and Delaunay can be considered. With the
best choice of approximation method, the performance of SFDs and GBD depends on
the property of the underlying surface. For the cases in which SFDs perform
better, the number of design points needed for SFDs is about half of or less
than that of the GBD to achieve the same prediction accuracy. SFDs that can be
tailored to high dimension and non-smooth surface are recommended especially
when large numbers of input factors need to be considered in the model.

Applications