title: MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
summary: Autoregressive transformers are spectacular models for short sequences but
  scale poorly to long sequences such as high-resolution images, podcasts, code,
  or books. We proposed Megabyte, a multi-scale decoder architecture that enables
  end-to-end differentiable modeling of sequences of over one million bytes.
  Megabyte segments sequences into patches and uses a local submodel within
  patches and a global model between patches. This enables sub-quadratic
  self-attention, much larger feedforward layers for the same compute, and
  improved parallelism during decoding -- unlocking better performance at reduced
  cost for both training and generation. Extensive experiments show that Megabyte
  allows byte-level models to perform competitively with subword models on long
  context language modeling, achieve state-of-the-art density estimation on
  ImageNet, and model audio from raw files. Together, these results establish the
  viability of tokenization-free autoregressive sequence modeling at scale.
paper_authors._id: 6461898256c57120a2d192e4
paper_authors.name: Lili Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T10:28:22.927Z
paper_authors._id: 6466d867f3e78d1d6be05da0
paper_authors.avatarUrl: /avatars/4a2458f9fe5e1637798adfaf01cfa4fa.svg
paper_authors.fullname: Lili Yu
paper_authors.type: user
paper_authors.user: lilisierrayu
paper_authors._id: 6461898256c57120a2d192e5
paper_authors.name: Dániel Simig
paper_authors._id: 6461898256c57120a2d192e6
paper_authors.name: Colin Flaherty
paper_authors._id: 6461898256c57120a2d192e7
paper_authors.name: Armen Aghajanyan
paper_authors._id: 6461898256c57120a2d192e8
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 6461898256c57120a2d192e9
paper_authors.name: Mike Lewis
paper_title: MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
paper_summary: Autoregressive transformers are spectacular models for short sequences but
  scale poorly to long sequences such as high-resolution images, podcasts, code,
  or books. We proposed Megabyte, a multi-scale decoder architecture that enables
  end-to-end differentiable modeling of sequences of over one million bytes.
  Megabyte segments sequences into patches and uses a local submodel within
  patches and a global model between patches. This enables sub-quadratic
  self-attention, much larger feedforward layers for the same compute, and
  improved parallelism during decoding -- unlocking better performance at reduced
  cost for both training and generation. Extensive experiments show that Megabyte
  allows byte-level models to perform competitively with subword models on long
  context language modeling, achieve state-of-the-art density estimation on
  ImageNet, and model audio from raw files. Together, these results establish the
  viability of tokenization-free autoregressive sequence modeling at scale.
paper_ai_keywords: autoregressive transformers multi-scale decoder architecture self-attention feedforward layers parallelism long context language modeling density estimation