title: Cheaply Evaluating Inference Efficiency Metrics for Autoregressive
    Transformer APIs
summary: Large language models (LLMs) power many state-of-the-art systems in natural
  language processing. However, these models are extremely computationally
  expensive, even at inference time, raising the natural question: when is the
  extra cost of deploying a larger model worth the anticipated boost in
  capabilities? Better understanding this tradeoff fundamentally could benefit
  from an inference efficiency metric that is both (i) easily comparable across
  models from different providers, and (ii) representative of the true cost of
  running queries in an isolated performance environment. Unfortunately, access
  to LLMs today is largely restricted to black-box text generation APIs and raw
  runtimes measured through this interface do not satisfy these desiderata: model
  providers can apply various software and hardware optimizations orthogonal to
  the model, and models served on shared infrastructure are susceptible to
  performance contention. To circumvent these problems, we propose a new metric
  for comparing inference efficiency across models. This metric puts models on
  equal footing as though they were served (i) on uniform hardware and software,
  and (ii) without performance contention. We call this metric the
  idealized runtime, and we propose a methodology to efficiently estimate
  this metric for autoregressive Transformer models. We also propose cost-aware
  variants that incorporate the number of accelerators needed to serve the model.
  Using these metrics, we compare ten state-of-the-art LLMs to provide the first
  analysis of inference efficiency-capability tradeoffs; we make several
  observations from this analysis, including the fact that the superior inference
  runtime performance of certain APIs is often a byproduct of optimizations
  within the API rather than the underlying model. Our methodology also
  facilitates the efficient comparison of different software and hardware stacks.
paper_authors._id: 6454604b363bb3aaf9ca58c5
paper_authors.name: Deepak Narayanan
paper_authors._id: 6454604b363bb3aaf9ca58c6
paper_authors.name: Keshav Santhanam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:36:31.278Z
paper_authors._id: 633328b6f695c3df38d97434
paper_authors.avatarUrl: /avatars/017e15cad6406653a9340cdbbee7dc51.svg
paper_authors.fullname: Keshav Santhanam
paper_authors.type: user
paper_authors.user: santhnm2
paper_authors._id: 6454604b363bb3aaf9ca58c7
paper_authors.name: Peter Henderson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:37:12.362Z
paper_authors._id: 617aafa1ff3db6021d069787
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644543485859-617aafa1ff3db6021d069787.jpeg
paper_authors.fullname: Peter Henderson
paper_authors.type: user
paper_authors.user: breakend
paper_authors._id: 6454604b363bb3aaf9ca58c8
paper_authors.name: Rishi Bommasani
paper_authors._id: 6454604b363bb3aaf9ca58c9
paper_authors.name: Tony Lee
paper_authors._id: 6454604b363bb3aaf9ca58ca
paper_authors.name: Percy Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:38:31.002Z
paper_authors._id: 6409651b9e9f790c905b2335
paper_authors.avatarUrl: /avatars/1fb8c80b60f21f65a0a027319101f236.svg
paper_authors.fullname: Percy Liang
paper_authors.type: user
paper_authors.user: percyliang
paper_title: Cheaply Evaluating Inference Efficiency Metrics for Autoregressive
    Transformer APIs
paper_summary: Large language models (LLMs) power many state-of-the-art systems in natural
  language processing. However, these models are extremely computationally
  expensive, even at inference time, raising the natural question: when is the
  extra cost of deploying a larger model worth the anticipated boost in
  capabilities? Better understanding this tradeoff fundamentally could benefit
  from an inference efficiency metric that is both (i) easily comparable across
  models from different providers, and (ii) representative of the true cost of
  running queries in an isolated performance environment. Unfortunately, access
  to LLMs today is largely restricted to black-box text generation APIs and raw
  runtimes measured through this interface do not satisfy these desiderata: model
  providers can apply various software and hardware optimizations orthogonal to
  the model, and models served on shared infrastructure are susceptible to
  performance contention. To circumvent these problems, we propose a new metric
  for comparing inference efficiency across models. This metric puts models on
  equal footing as though they were served (i) on uniform hardware and software,
  and (ii) without performance contention. We call this metric the
  idealized runtime, and we propose a methodology to efficiently estimate
  this metric for autoregressive Transformer models. We also propose cost-aware
  variants that incorporate the number of accelerators needed to serve the model.
  Using these metrics, we compare ten state-of-the-art LLMs to provide the first
  analysis of inference efficiency-capability tradeoffs; we make several
  observations from this analysis, including the fact that the superior inference
  runtime performance of certain APIs is often a byproduct of optimizations
  within the API rather than the underlying model. Our methodology also
  facilitates the efficient comparison of different software and hardware stacks.
paper_ai_summary: A new metric called idealized runtime is proposed to compare inference efficiency across large language models, facilitating analysis of tradeoffs between runtime performance and model capabilities.
paper_ai_keywords: Large language models LLMs natural language processing inference efficiency Transformer models autoregressive models idealized runtime cost-aware variants