  Rising computational demands of modern natural language processing (NLP)
systems have increased the barrier to entry for cutting-edge research while
posing serious environmental concerns. Yet, progress on model efficiency has
been impeded by practical challenges in model evaluation and comparison. For
example, hardware is challenging to control due to disparate levels of
accessibility across different institutions. Moreover, improvements in metrics
such as FLOPs often fail to translate to progress in real-world applications.
In response, we introduce Pentathlon, a benchmark for holistic and realistic
evaluation of model efficiency. Pentathlon focuses on inference, which accounts
for a majority of the compute in a model's lifecycle. It offers a
strictly-controlled hardware platform, and is designed to mirror real-world
applications scenarios. It incorporates a suite of metrics that target
different aspects of efficiency, including latency, throughput, memory
overhead, and energy consumption. Pentathlon also comes with a software library
that can be seamlessly integrated into any codebase and enable evaluation. As a
standardized and centralized evaluation platform, Pentathlon can drastically
reduce the workload to make fair and reproducible efficiency comparisons. While
initially focused on natural language processing (NLP) models, Pentathlon is
designed to allow flexible extension to other fields. We envision Pentathlon
will stimulate algorithmic innovations in building efficient models, and foster
an increased awareness of the social and environmental implications in the
development of future-generation NLP models.

Computation and Language