title: The Languini Kitchen: Enabling Language Modelling Research at Different
    Scales of Compute
summary: The Languini Kitchen serves as both a research collective and codebase
  designed to empower researchers with limited computational resources to
  contribute meaningfully to the field of language modelling. We introduce an
  experimental protocol that enables model comparisons based on equivalent
  compute, measured in accelerator hours. The number of tokens on which a model
  is trained is defined by the model's throughput and the chosen compute class.
  Notably, this approach avoids constraints on critical hyperparameters which
  affect total parameters or floating-point operations. For evaluation, we
  pre-process an existing large, diverse, and high-quality dataset of books that
  surpasses existing academic benchmarks in quality, diversity, and document
  length. On it, we compare methods based on their empirical scaling trends which
  are estimated through experiments at various levels of compute. This work also
  provides two baseline models: a feed-forward model derived from the GPT-2
  architecture and a recurrent model in the form of a novel LSTM with ten-fold
  throughput. While the GPT baseline achieves better perplexity throughout all
  our levels of compute, our LSTM baseline exhibits a predictable and more
  favourable scaling law. This is due to the improved throughput and the need for
  fewer training tokens to achieve the same decrease in test perplexity.
  Extrapolating the scaling laws leads of both models results in an intersection
  at roughly 50,000 accelerator hours. We hope this work can serve as the
  foundation for meaningful and reproducible language modelling research.
paper_authors._id: 650bd3f00f4a84a5db565e92
paper_authors.name: Aleksandar Stanić
paper_authors._id: 650bd3f00f4a84a5db565e93
paper_authors.name: Dylan Ashley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T10:02:06.175Z
paper_authors._id: 63ff35c44852102d4873de7d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677669808298-noauth.png
paper_authors.fullname: Dylan Ashley
paper_authors.type: user
paper_authors.user: dylanashley
paper_authors._id: 650bd3f00f4a84a5db565e94
paper_authors.name: Oleg Serikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T10:02:13.749Z
paper_authors._id: 614d9b7e398ee2c4c1274f2e
paper_authors.avatarUrl: /avatars/ace33cf5b77e81eb7702a27ee53c4000.svg
paper_authors.fullname: Oleg Serikov
paper_authors.type: user
paper_authors.user: oserikov
paper_authors._id: 650bd3f00f4a84a5db565e95
paper_authors.name: Louis Kirsch
paper_authors._id: 650bd3f00f4a84a5db565e96
paper_authors.name: Francesco Faccio
paper_authors._id: 650bd3f00f4a84a5db565e97
paper_authors.name: Jürgen Schmidhuber
paper_authors._id: 650bd3f00f4a84a5db565e98
paper_authors.name: Thomas Hofmann
paper_authors._id: 650bd3f00f4a84a5db565e99
paper_authors.name: Imanol Schlag
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-09-21T05:26:10.064Z
paper_authors._id: 6310fac907a768279024d158
paper_authors.avatarUrl: /avatars/5bced71b6d507e621ec159ada94480f0.svg
paper_authors.fullname: Imanol Schlag
paper_authors.type: user
paper_authors.user: ischlag
paper_title: The Languini Kitchen: Enabling Language Modelling Research at Different
    Scales of Compute
paper_summary: The Languini Kitchen serves as both a research collective and codebase
  designed to empower researchers with limited computational resources to
  contribute meaningfully to the field of language modelling. We introduce an
  experimental protocol that enables model comparisons based on equivalent
  compute, measured in accelerator hours. The number of tokens on which a model
  is trained is defined by the model's throughput and the chosen compute class.
  Notably, this approach avoids constraints on critical hyperparameters which
  affect total parameters or floating-point operations. For evaluation, we
  pre-process an existing large, diverse, and high-quality dataset of books that
  surpasses existing academic benchmarks in quality, diversity, and document
  length. On it, we compare methods based on their empirical scaling trends which
  are estimated through experiments at various levels of compute. This work also
  provides two baseline models: a feed-forward model derived from the GPT-2
  architecture and a recurrent model in the form of a novel LSTM with ten-fold
  throughput. While the GPT baseline achieves better perplexity throughout all
  our levels of compute, our LSTM baseline exhibits a predictable and more
  favourable scaling law. This is due to the improved throughput and the need for
  fewer training tokens to achieve the same decrease in test perplexity.
  Extrapolating the scaling laws leads of both models results in an intersection
  at roughly 50,000 accelerator hours. We hope this work can serve as the
  foundation for meaningful and reproducible language modelling research.
paper_ai_summary: A protocol for comparing language models based on equivalent computational resources evaluates GPT-2 and a new high-throughput LSTM, showing predictable scaling and intersection in performance at 50,000 accelerator hours.
paper_ai_keywords: feed-forward model GPT-2 architecture recurrent model LSTM perplexity throughput empirical scaling trends accelerator hours training tokens document length