Training LLMs over Neurally Compressed Text
In this paper, we explore the idea of training large language models (LLMs)
over highly compressed text. While standard subword tokenizers compress text by
a small factor, neural text compressors can achieve much higher rates of
compression. If it were possible to train LLMs directly over neurally
compressed text, this would confer advantages in training and serving
efficiency, as well as easier handling of long text spans. The main obstacle to
this goal is that strong compression tends to produce opaque outputs that are
not well-suited for learning. In particular, we find that text na\"ively
compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome
this, we propose Equal-Info Windows, a novel compression technique whereby text
is segmented into blocks that each compress to the same bit length. Using this
method, we demonstrate effective learning over neurally compressed text that
improves with scale, and outperforms byte-level baselines by a wide margin on
perplexity and inference speed benchmarks. While our method delivers worse
perplexity than subword tokenizers for models trained with the same parameter
count, it has the benefit of shorter sequence lengths. Shorter sequence lengths
require fewer autoregressive generation steps, and reduce latency. Finally, we
provide extensive analysis of the properties that contribute to learnability,
and offer concrete suggestions for how to further improve the performance of
high-compression tokenizers.
660f6519bbc30647f1684305
Brian Lester
660f6519bbc30647f1684306
Jaehoon Lee
660f6519bbc30647f1684307
Alex Alemi
660f6519bbc30647f1684308
Jeffrey Pennington
660f6519bbc30647f1684309
Adam Roberts
660f6519bbc30647f168430a
Jascha Sohl-Dickstein
660f6519bbc30647f168430b
Noah Constant
Training LLMs over Neurally Compressed Text
In this paper, we explore the idea of training large language models (LLMs)
over highly compressed text. While standard subword tokenizers compress text by
a small factor, neural text compressors can achieve much higher rates of
compression. If it were possible to train LLMs directly over neurally
compressed text, this would confer advantages in training and serving
efficiency, as well as easier handling of long text spans. The main obstacle to
this goal is that strong compression tends to produce opaque outputs that are
not well-suited for learning. In particular, we find that text na\"ively
compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome
this, we propose Equal-Info Windows, a novel compression technique whereby text
is segmented into blocks that each compress to the same bit length. Using this
method, we demonstrate effective learning over neurally compressed text that
improves with scale, and outperforms byte-level baselines by a wide margin on
perplexity and inference speed benchmarks. While our method delivers worse
perplexity than subword tokenizers for models trained with the same parameter
count, it has the benefit of shorter sequence lengths. Shorter sequence lengths
require fewer autoregressive generation steps, and reduce latency. Finally, we
provide extensive analysis of the properties that contribute to learnability,
and offer concrete suggestions for how to further improve the performance of
high-compression tokenizers.
Equal-Info Windows compress text uniformly for effective LLM training, reducing sequence lengths and latency.
large language models LLMs neural text compressors Arithmetic Coding Equal-Info Windows perplexity inference speed subword tokenizers autoregressive generation high-compression tokenizers