title: Language Modeling Is Compression
summary: It has long been established that predictive models can be transformed into
  lossless compressors and vice versa. Incidentally, in recent years, the machine
  learning community has focused on training increasingly large and powerful
  self-supervised (language) models. Since these large language models exhibit
  impressive predictive capabilities, they are well-positioned to be strong
  compressors. In this work, we advocate for viewing the prediction problem
  through the lens of compression and evaluate the compression capabilities of
  large (foundation) models. We show that large language models are powerful
  general-purpose predictors and that the compression viewpoint provides novel
  insights into scaling laws, tokenization, and in-context learning. For example,
  Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to
  43.4% and LibriSpeech samples to 16.4% of their raw size, beating
  domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively.
  Finally, we show that the prediction-compression equivalence allows us to use
  any compressor (like gzip) to build a conditional generative model.
paper_authors._id: 650a50b9d48ed98e63cebb86
paper_authors.name: Grégoire Delétang
paper_authors._id: 650a50b9d48ed98e63cebb87
paper_authors.name: Anian Ruoss
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:06:43.296Z
paper_authors._id: 65c51758522ee3a4edd917d0
paper_authors.avatarUrl: /avatars/2ea3b64b5eeb34e2bfe821e4210f1b3e.svg
paper_authors.fullname: Anian Ruoss
paper_authors.type: user
paper_authors.user: anianruoss
paper_authors._id: 650a50b9d48ed98e63cebb88
paper_authors.name: Paul-Ambroise Duquenne
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T10:28:38.106Z
paper_authors._id: 6331a09fe092098b57bc1d83
paper_authors.avatarUrl: /avatars/cdbc4be17c160b5e1f957b6c1ae29e22.svg
paper_authors.fullname: Paul-Ambroise Duquenne
paper_authors.type: user
paper_authors.user: padqn
paper_authors._id: 650a50b9d48ed98e63cebb89
paper_authors.name: Elliot Catt
paper_authors._id: 650a50b9d48ed98e63cebb8a
paper_authors.name: Tim Genewein
paper_authors._id: 650a50b9d48ed98e63cebb8b
paper_authors.name: Christopher Mattern
paper_authors._id: 650a50b9d48ed98e63cebb8c
paper_authors.name: Jordi Grau-Moya
paper_authors._id: 650a50b9d48ed98e63cebb8d
paper_authors.name: Li Kevin Wenliang
paper_authors._id: 650a50b9d48ed98e63cebb8e
paper_authors.name: Matthew Aitchison
paper_authors._id: 650a50b9d48ed98e63cebb8f
paper_authors.name: Laurent Orseau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:28:01.174Z
paper_authors._id: 64ae74ad71b7e849b0aa3404
paper_authors.avatarUrl: /avatars/50585fdd3fc2a916814c500ca6bb3c00.svg
paper_authors.fullname: Laurent Orseau
paper_authors.type: user
paper_authors.user: lorseau
paper_authors._id: 650a50b9d48ed98e63cebb90
paper_authors.name: Marcus Hutter
paper_authors._id: 650a50b9d48ed98e63cebb91
paper_authors.name: Joel Veness
paper_title: Language Modeling Is Compression
paper_summary: It has long been established that predictive models can be transformed into
  lossless compressors and vice versa. Incidentally, in recent years, the machine
  learning community has focused on training increasingly large and powerful
  self-supervised (language) models. Since these large language models exhibit
  impressive predictive capabilities, they are well-positioned to be strong
  compressors. In this work, we advocate for viewing the prediction problem
  through the lens of compression and evaluate the compression capabilities of
  large (foundation) models. We show that large language models are powerful
  general-purpose predictors and that the compression viewpoint provides novel
  insights into scaling laws, tokenization, and in-context learning. For example,
  Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to
  43.4% and LibriSpeech samples to 16.4% of their raw size, beating
  domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively.
  Finally, we show that the prediction-compression equivalence allows us to use
  any compressor (like gzip) to build a conditional generative model.
paper_ai_keywords: self-supervised models large (foundation) models predictive capabilities compression viewpoint scaling laws tokenization in-context learning Chinchilla 70B ImageNet patches LibriSpeech samples domain-specific compressors PNG FLAC conditional generative model