title: Training LLMs over Neurally Compressed Text
summary: In this paper, we explore the idea of training large language models (LLMs)
  over highly compressed text. While standard subword tokenizers compress text by
  a small factor, neural text compressors can achieve much higher rates of
  compression. If it were possible to train LLMs directly over neurally
  compressed text, this would confer advantages in training and serving
  efficiency, as well as easier handling of long text spans. The main obstacle to
  this goal is that strong compression tends to produce opaque outputs that are
  not well-suited for learning. In particular, we find that text na\"ively
  compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome
  this, we propose Equal-Info Windows, a novel compression technique whereby text
  is segmented into blocks that each compress to the same bit length. Using this
  method, we demonstrate effective learning over neurally compressed text that
  improves with scale, and outperforms byte-level baselines by a wide margin on
  perplexity and inference speed benchmarks. While our method delivers worse
  perplexity than subword tokenizers for models trained with the same parameter
  count, it has the benefit of shorter sequence lengths. Shorter sequence lengths
  require fewer autoregressive generation steps, and reduce latency. Finally, we
  provide extensive analysis of the properties that contribute to learnability,
  and offer concrete suggestions for how to further improve the performance of
  high-compression tokenizers.
paper_authors._id: 660f6519bbc30647f1684305
paper_authors.name: Brian Lester
paper_authors._id: 660f6519bbc30647f1684306
paper_authors.name: Jaehoon Lee
paper_authors._id: 660f6519bbc30647f1684307
paper_authors.name: Alex Alemi
paper_authors._id: 660f6519bbc30647f1684308
paper_authors.name: Jeffrey Pennington
paper_authors._id: 660f6519bbc30647f1684309
paper_authors.name: Adam Roberts
paper_authors._id: 660f6519bbc30647f168430a
paper_authors.name: Jascha Sohl-Dickstein
paper_authors._id: 660f6519bbc30647f168430b
paper_authors.name: Noah Constant
paper_title: Training LLMs over Neurally Compressed Text
paper_summary: In this paper, we explore the idea of training large language models (LLMs)
  over highly compressed text. While standard subword tokenizers compress text by
  a small factor, neural text compressors can achieve much higher rates of
  compression. If it were possible to train LLMs directly over neurally
  compressed text, this would confer advantages in training and serving
  efficiency, as well as easier handling of long text spans. The main obstacle to
  this goal is that strong compression tends to produce opaque outputs that are
  not well-suited for learning. In particular, we find that text na\"ively
  compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome
  this, we propose Equal-Info Windows, a novel compression technique whereby text
  is segmented into blocks that each compress to the same bit length. Using this
  method, we demonstrate effective learning over neurally compressed text that
  improves with scale, and outperforms byte-level baselines by a wide margin on
  perplexity and inference speed benchmarks. While our method delivers worse
  perplexity than subword tokenizers for models trained with the same parameter
  count, it has the benefit of shorter sequence lengths. Shorter sequence lengths
  require fewer autoregressive generation steps, and reduce latency. Finally, we
  provide extensive analysis of the properties that contribute to learnability,
  and offer concrete suggestions for how to further improve the performance of
  high-compression tokenizers.
paper_ai_summary: Equal-Info Windows compress text uniformly for effective LLM training, reducing sequence lengths and latency.
paper_ai_keywords: large language models LLMs neural text compressors Arithmetic Coding Equal-Info Windows perplexity inference speed subword tokenizers autoregressive generation high-compression tokenizers