MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Autoregressive transformers are spectacular models for short sequences but
scale poorly to long sequences such as high-resolution images, podcasts, code,
or books. We proposed Megabyte, a multi-scale decoder architecture that enables
end-to-end differentiable modeling of sequences of over one million bytes.
Megabyte segments sequences into patches and uses a local submodel within
patches and a global model between patches. This enables sub-quadratic
self-attention, much larger feedforward layers for the same compute, and
improved parallelism during decoding -- unlocking better performance at reduced
cost for both training and generation. Extensive experiments show that Megabyte
allows byte-level models to perform competitively with subword models on long
context language modeling, achieve state-of-the-art density estimation on
ImageNet, and model audio from raw files. Together, these results establish the
viability of tokenization-free autoregressive sequence modeling at scale.
6461898256c57120a2d192e4
Lili Yu
claimed_verified
2023-05-19T10:28:22.927Z
6466d867f3e78d1d6be05da0
/avatars/4a2458f9fe5e1637798adfaf01cfa4fa.svg
Lili Yu
user
lilisierrayu
6461898256c57120a2d192e5
Dániel Simig
6461898256c57120a2d192e6
Colin Flaherty
6461898256c57120a2d192e7
Armen Aghajanyan
6461898256c57120a2d192e8
Luke Zettlemoyer
6461898256c57120a2d192e9
Mike Lewis
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Autoregressive transformers are spectacular models for short sequences but
scale poorly to long sequences such as high-resolution images, podcasts, code,
or books. We proposed Megabyte, a multi-scale decoder architecture that enables
end-to-end differentiable modeling of sequences of over one million bytes.
Megabyte segments sequences into patches and uses a local submodel within
patches and a global model between patches. This enables sub-quadratic
self-attention, much larger feedforward layers for the same compute, and
improved parallelism during decoding -- unlocking better performance at reduced
cost for both training and generation. Extensive experiments show that Megabyte
allows byte-level models to perform competitively with subword models on long
context language modeling, achieve state-of-the-art density estimation on
ImageNet, and model audio from raw files. Together, these results establish the
viability of tokenization-free autoregressive sequence modeling at scale.
Megabyte, a multi-scale decoder architecture, enables efficient byte-level modeling of long sequences through sub-quadratic self-attention, larger feedforward layers, and improved parallelism, achieving performance competitive with subword models and state-of-the-art results.
Megabyte multi-scale decoder architecture sub-quadratic self-attention feedforward layers parallelism long context language modeling density estimation ImageNet autoregressive sequence modeling tokenization-free