The world sees a proliferation of machine learning/deep learning (ML) models and their wide adoption in different application domainsrecently . This has made the profiling and characterization of ML models an increasingly pressing task for both hardware designers and system providers, as they would like to offer the best possible computing system to serve ML models with the desired latency, throughput,  and energy requirements while maximizing resource utilization. Such an endeavor is challenging as the characteristics of an ML model depend on the interplay between the model, framework, system libraries, and the hardware (or the HW/SW stack). A thorough characterization requires understanding the behavior of the model execution across the HW/SW stack levels. Existing profiling tools are disjoint, however, and only focus on profiling within a particular level of the stack  . This paper proposes a leveled profiling design that leverages existing profiling tools to perform across-stack profiling. The design does so in spite of the profiling overheads incurred from the profiling providers. We coupled the profiling capability with an automatic analysis pipeline to systematically characterize 65 state-of-the-art ML models. Through this characterization, we show that our across-stack profiling solution provides insights (which are difficult to discern otherwise ) on the characteristics of ML models, ML frameworks, and GPU hardware .
There has been a rapid proliferation of machine learning/deep learning (ML) models and wide adoption of them in many application domains . This has made profiling and characterization of ML model performance an increasingly pressing task for both hardware designers and system providers, as they would like to offer the best possible system to serve ML models with the target latency, throughput, cost, and energy requirements while maximizing resource utilization. Such an endeavor is challenging as the characteristics of an ML model depend on the interplay between the model, framework, system libraries, and the hardware (or the HW/SW stack). Existing profiling tools are disjoint, however, and only focus on profiling within a particular level of the stack , which limits the thoroughness and usefulness of the profiling results . This paper proposes XSP --- an across-stack profiling design that gives a holistic and hierarchical view of ML model execution. XSP leverages distributed tracing to aggregate and correlate profile data from different sources. XSP introduces a leveled and iterative measurement approach that accurately captures the latencies at all levels of the HW/SW stack in spite of the profiling overhead. We couple the profiling design with an automated analysis pipeline to systematically analyze 65 state-of-the-art ML models. We demonstrate that XSP provides insights which would be difficult to discern otherwise .
R
The world sees a
There has been a rapid
R
their wide adoption in different application domainsrecently
wide adoption of them in many application domains
D
the
R
models
model performance
D
computing
R
desired
target
A
cost,
D
A thorough characterization requires understanding the behavior of the model execution across the HW/SW stack levels.
A
, which limits the thoroughness and usefulness of the profiling results
R
a leveled
XSP --- an across-stack
R
leverages existing profiling tools to perform across-stack profiling. The design does so
gives a holistic and hierarchical view of ML model execution. XSP leverages distributed tracing to aggregate and correlate profile data from different sources. XSP introduces a leveled and iterative measurement approach that accurately captures the latencies at all levels of the HW/SW stack
R
overheads incurred from the profiling providers. We coupled the profiling capability with an automatic
overhead. We couple the profiling design with an automated
R
characterize
analyze
R
Through this characterization, we show that our across-stack profiling solution provides insights (which are
We demonstrate that XSP provides insights which would be
D
) on the characteristics of ML models, ML frameworks, and GPU hardware