title: Byte Latent Transformer: Patches Scale Better Than Tokens
summary: We introduce the Byte Latent Transformer (BLT), a new byte-level LLM
  architecture that, for the first time, matches tokenization-based LLM
  performance at scale with significant improvements in inference efficiency and
  robustness. BLT encodes bytes into dynamically sized patches, which serve as
  the primary units of computation. Patches are segmented based on the entropy of
  the next byte, allocating more compute and model capacity where increased data
  complexity demands it. We present the first FLOP controlled scaling study of
  byte-level models up to 8B parameters and 4T training bytes. Our results
  demonstrate the feasibility of scaling models trained on raw bytes without a
  fixed vocabulary. Both training and inference efficiency improve due to
  dynamically selecting long patches when data is predictable, along with
  qualitative improvements on reasoning and long tail generalization. Overall,
  for fixed inference costs, BLT shows significantly better scaling than
  tokenization-based models, by simultaneously growing both patch and model size.
paper_authors._id: 675ffc2a79e2036a7090d4ff
paper_authors.name: Artidoro Pagnoni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:12.348Z
paper_authors._id: 6279d059812ee439d9c8057c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6279d059812ee439d9c8057c/2E3-rFQganxuz3jDL-Esc.jpeg
paper_authors.fullname: Artidoro Pagnoni
paper_authors.type: user
paper_authors.user: artidoro
paper_authors._id: 675ffc2a79e2036a7090d500
paper_authors.name: Ram Pasunuru
paper_authors._id: 675ffc2a79e2036a7090d501
paper_authors.name: Pedro Rodriguez
paper_authors._id: 675ffc2a79e2036a7090d502
paper_authors.name: John Nguyen
paper_authors._id: 675ffc2a79e2036a7090d503
paper_authors.name: Benjamin Muller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:24:59.088Z
paper_authors._id: 5e6b7153d4cd9779932a7600
paper_authors.avatarUrl: /avatars/c1304f100758c5e2a54fd4d47f638841.svg
paper_authors.fullname: Benjamin Muller
paper_authors.type: user
paper_authors.user: benjamin-mlr
paper_authors._id: 675ffc2a79e2036a7090d504
paper_authors.name: Margaret Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:24:38.264Z
paper_authors._id: 66edd75d5718c003b9c66c09
paper_authors.avatarUrl: /avatars/2bc2067b0c2a61092e401c3237921f19.svg
paper_authors.fullname: Margaret Li
paper_authors.type: user
paper_authors.user: marg33
paper_authors._id: 675ffc2a79e2036a7090d505
paper_authors.name: Chunting Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:24:30.461Z
paper_authors._id: 64f8d07dd493d8b0d2c04ecd
paper_authors.avatarUrl: /avatars/7112232dcadcd2a7c12ec635d781b3ea.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Chunting
paper_authors._id: 675ffc2a79e2036a7090d506
paper_authors.name: Lili Yu
paper_authors._id: 675ffc2a79e2036a7090d507
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:24:09.875Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 675ffc2a79e2036a7090d508
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 675ffc2a79e2036a7090d509
paper_authors.name: Gargi Ghosh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:24:01.190Z
paper_authors._id: 664767349e82dfd4a36db460
paper_authors.avatarUrl: /avatars/df9af7bdc1d5117571d1b2349feade44.svg
paper_authors.fullname: Gargi Ghosh
paper_authors.type: user
paper_authors.user: gargighosh
paper_authors._id: 675ffc2a79e2036a7090d50a
paper_authors.name: Mike Lewis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:23:55.718Z
paper_authors._id: 5e67eda71009063689407484
paper_authors.avatarUrl: /avatars/6d7688faba888d463584eaa02ad011e3.svg
paper_authors.fullname: Mike Lewis
paper_authors.type: user
paper_authors.user: mikelewis0
paper_authors._id: 675ffc2a79e2036a7090d50b
paper_authors.name: Ari Holtzman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:23:40.460Z
paper_authors._id: 6508c81db44445e9b36f3a1e
paper_authors.avatarUrl: /avatars/7d4b257398c601b482b9655f046021f3.svg
paper_authors.fullname: Ari Holtzman
paper_authors.type: user
paper_authors.user: ariholtzman
paper_authors._id: 675ffc2a79e2036a7090d50c
paper_authors.name: Srinivasan Iyer
paper_title: Byte Latent Transformer: Patches Scale Better Than Tokens
paper_summary: We introduce the Byte Latent Transformer (BLT), a new byte-level LLM
  architecture that, for the first time, matches tokenization-based LLM
  performance at scale with significant improvements in inference efficiency and
  robustness. BLT encodes bytes into dynamically sized patches, which serve as
  the primary units of computation. Patches are segmented based on the entropy of
  the next byte, allocating more compute and model capacity where increased data
  complexity demands it. We present the first FLOP controlled scaling study of
  byte-level models up to 8B parameters and 4T training bytes. Our results
  demonstrate the feasibility of scaling models trained on raw bytes without a
  fixed vocabulary. Both training and inference efficiency improve due to
  dynamically selecting long patches when data is predictable, along with
  qualitative improvements on reasoning and long tail generalization. Overall,
  for fixed inference costs, BLT shows significantly better scaling than
  tokenization-based models, by simultaneously growing both patch and model size.
paper_ai_summary: A Byte Latent Transformer (BLT) matches tokenization-based LLM performance at scale with improved inference efficiency and robustness, using entropy-based byte patching.
paper_ai_keywords: Byte Latent Transformer (BLT) byte-level LLM dynamically sized patches entropy FLOP controlled scaling byte-level models model capacity long patches reasoning long tail generalization