From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Tokenization imposes a fixed granularity on the input text, freezing how a
language model operates on data and how far in the future it predicts. Byte
Pair Encoding (BPE) and similar schemes split text once, build a static
vocabulary, and leave the model stuck with that choice. We relax this rigidity
by introducing an autoregressive U-Net that learns to embed its own tokens as
it trains. The network reads raw bytes, pools them into words, then pairs of
words, then up to 4 words, giving it a multi-scale view of the sequence. At
deeper stages, the model must predict further into the future -- anticipating
the next few words rather than the next byte -- so deeper stages focus on
broader semantic patterns while earlier stages handle fine details. When
carefully tuning and controlling pretraining compute, shallow hierarchies tie
strong BPE baselines, and deeper hierarchies have a promising trend. Because
tokenization now lives inside the model, the same system can handle
character-level tasks and carry knowledge across low-resource languages.
685264ca0164cd13167105f2
Mathurin Videau
685264ca0164cd13167105f3
Badr Youbi Idrissi
685264ca0164cd13167105f4
Alessandro Leite
685264ca0164cd13167105f5
Marc Schoenauer
685264ca0164cd13167105f6
Olivier Teytaud
685264ca0164cd13167105f7
David Lopez-Paz
From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Tokenization imposes a fixed granularity on the input text, freezing how a
language model operates on data and how far in the future it predicts. Byte
Pair Encoding (BPE) and similar schemes split text once, build a static
vocabulary, and leave the model stuck with that choice. We relax this rigidity
by introducing an autoregressive U-Net that learns to embed its own tokens as
it trains. The network reads raw bytes, pools them into words, then pairs of
words, then up to 4 words, giving it a multi-scale view of the sequence. At
deeper stages, the model must predict further into the future -- anticipating
the next few words rather than the next byte -- so deeper stages focus on
broader semantic patterns while earlier stages handle fine details. When
carefully tuning and controlling pretraining compute, shallow hierarchies tie
strong BPE baselines, and deeper hierarchies have a promising trend. Because
tokenization now lives inside the model, the same system can handle
character-level tasks and carry knowledge across low-resource languages.
An autoregressive U-Net learns to embed its own tokens during training, enabling a multi-scale view of text sequences and improved handling of character-level tasks and low-resource languages.
autoregressive U-Net Byte Pair Encoding (BPE) multi-scale view semantic patterns character-level tasks low-resource languages