  Cloud computing and virtualization solutions allow one to rent the virtual
machines (VMs) needed to run applications on a pay-per-use basis, but rented
VMs do not offer any guarantee on their performance. Cloud platforms are known
to be affected by performance variability, but a better understanding is still
required. This paper moves in that direction and presents an in-depth,
multi-faceted study on the performance variability of VMs. Unlike previous
studies, our assessment covers a wide range of factors: 16 VM types from 4
well-known cloud providers, 10 benchmarks, and 28 different metrics. We present
four new contributions. First, we introduce a new benchmark suite (VMBS) that
let researchers and practitioners systematically collect a diverse set of
performance data. Second, we present a new indicator, called Variability
Indicator, that allows for measuring variability in the performance of VMs.
Third, we illustrate an analysis of the collected data across four different
dimensions: resources, isolation, time, and cost. Fourth, we present multiple
predictive models based on Machine Learning that aim to forecast future
performance and detect time patterns. Our experiments provide important
insights on the resource variability of VMs, highlighting differences and
similarities between various cloud providers. To the best of our knowledge,
this is the widest analysis ever conducted on the topic.

Software Engineering;Distributed, Parallel, and Cluster Computing