title: Benchmarking Large Language Model Capabilities for Conditional
    Generation
summary: Pre-trained large language models (PLMs) underlie most new developments in
  natural language processing. They have shifted the field from
  application-specific model pipelines to a single model that is adapted to a
  wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside
  techniques like few-shot learning, have additionally shifted the output
  modality to generation instead of classification or regression. Despite their
  ubiquitous use, the generation quality of language models is rarely evaluated
  when these models are introduced. Additionally, it is unclear how existing
  generation tasks--while they can be used to compare systems at a high
  level--relate to the real world use cases for which people have been adopting
  them. In this work, we discuss how to adapt existing application-specific
  generation benchmarks to PLMs and provide an in-depth, empirical study of the
  limitations and capabilities of PLMs in natural language generation tasks along
  dimensions such as scale, architecture, input and output language. Our results
  show that PLMs differ in their applicability to different data regimes and
  their generalization to multiple languages and inform which PLMs to use for a
  given generation task setup. We share best practices to be taken into
  consideration when benchmarking generation capabilities during the development
  of upcoming PLMs.
paper_authors._id: 649e3d76a8314342768b87c2
paper_authors.name: Joshua Maynez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-30T20:55:58.479Z
paper_authors._id: 63f5fd44b607296857b3d059
paper_authors.avatarUrl: /avatars/7ea62ff0d82e24edfbc7a562484d0947.svg
paper_authors.fullname: Joshua Maynez
paper_authors.type: user
paper_authors.user: joshuahm
paper_authors._id: 649e3d76a8314342768b87c3
paper_authors.name: Priyanka Agrawal
paper_authors._id: 649e3d76a8314342768b87c4
paper_authors.name: Sebastian Gehrmann
paper_title: Benchmarking Large Language Model Capabilities for Conditional
    Generation
paper_summary: Pre-trained large language models (PLMs) underlie most new developments in
  natural language processing. They have shifted the field from
  application-specific model pipelines to a single model that is adapted to a
  wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside
  techniques like few-shot learning, have additionally shifted the output
  modality to generation instead of classification or regression. Despite their
  ubiquitous use, the generation quality of language models is rarely evaluated
  when these models are introduced. Additionally, it is unclear how existing
  generation tasks--while they can be used to compare systems at a high
  level--relate to the real world use cases for which people have been adopting
  them. In this work, we discuss how to adapt existing application-specific
  generation benchmarks to PLMs and provide an in-depth, empirical study of the
  limitations and capabilities of PLMs in natural language generation tasks along
  dimensions such as scale, architecture, input and output language. Our results
  show that PLMs differ in their applicability to different data regimes and
  their generalization to multiple languages and inform which PLMs to use for a
  given generation task setup. We share best practices to be taken into
  consideration when benchmarking generation capabilities during the development
  of upcoming PLMs.
paper_ai_summary: Researchers adapt application-specific generation benchmarks to pre-trained large language models (PLMs) to evaluate their limitations and capabilities across different dimensions like scale, architecture, and language.
paper_ai_keywords: pre-trained large language models PLMs natural language processing autoregressive PLMs few-shot learning generation classification regression benchmarking natural language generation