  In recent years, several HPC facilities have started continuous monitoring of
their systems and jobs to collect performance-related data for understanding
performance and operational efficiency. Such data can be used to optimize the
performance of individual jobs and the overall system by creating data-driven
models that can predict the performance of pending jobs. In this paper, we
model the performance of representative control jobs using longitudinal
system-wide monitoring data to explore the causes of performance variability.
Using machine learning, we are able to predict the performance of unseen jobs
before they are executed based on the current system state. We analyze these
prediction models in great detail to identify the features that are dominant
predictors of performance. We demonstrate that such models can be
application-agnostic and can be used for predicting performance of applications
that are not included in training.

Distributed, Parallel, and Cluster Computing;Machine Learning;Performance