Byte Latent Transformer: Patches Scale Better Than Tokens
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM
architecture that, for the first time, matches tokenization-based LLM
performance at scale with significant improvements in inference efficiency and
robustness. BLT encodes bytes into dynamically sized patches, which serve as
the primary units of computation. Patches are segmented based on the entropy of
the next byte, allocating more compute and model capacity where increased data
complexity demands it. We present the first FLOP controlled scaling study of
byte-level models up to 8B parameters and 4T training bytes. Our results
demonstrate the feasibility of scaling models trained on raw bytes without a
fixed vocabulary. Both training and inference efficiency improve due to
dynamically selecting long patches when data is predictable, along with
qualitative improvements on reasoning and long tail generalization. Overall,
for fixed inference costs, BLT shows significantly better scaling than
tokenization-based models, by simultaneously growing both patch and model size.
675ffc2a79e2036a7090d4ff
Artidoro Pagnoni
claimed_verified
2024-12-17T08:04:12.348Z
6279d059812ee439d9c8057c
https://cdn-avatars.huggingface.co/v1/production/uploads/6279d059812ee439d9c8057c/2E3-rFQganxuz3jDL-Esc.jpeg
Artidoro Pagnoni
user
artidoro
675ffc2a79e2036a7090d500
Ram Pasunuru
675ffc2a79e2036a7090d501
Pedro Rodriguez
675ffc2a79e2036a7090d502
John Nguyen
675ffc2a79e2036a7090d503
Benjamin Muller
admin_assigned
2024-12-17T11:24:59.088Z
5e6b7153d4cd9779932a7600
/avatars/c1304f100758c5e2a54fd4d47f638841.svg
Benjamin Muller
user
benjamin-mlr
675ffc2a79e2036a7090d504
Margaret Li
admin_assigned
2024-12-17T11:24:38.264Z
66edd75d5718c003b9c66c09
/avatars/2bc2067b0c2a61092e401c3237921f19.svg
Margaret Li
user
marg33
675ffc2a79e2036a7090d505
Chunting Zhou
admin_assigned
2024-12-17T11:24:30.461Z
64f8d07dd493d8b0d2c04ecd
/avatars/7112232dcadcd2a7c12ec635d781b3ea.svg
Zhou
user
Chunting
675ffc2a79e2036a7090d506
Lili Yu
675ffc2a79e2036a7090d507
Jason Weston
admin_assigned
2024-12-17T11:24:09.875Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
675ffc2a79e2036a7090d508
Luke Zettlemoyer
675ffc2a79e2036a7090d509
Gargi Ghosh
admin_assigned
2024-12-17T11:24:01.190Z
664767349e82dfd4a36db460
/avatars/df9af7bdc1d5117571d1b2349feade44.svg
Gargi Ghosh
user
gargighosh
675ffc2a79e2036a7090d50a
Mike Lewis
admin_assigned
2024-12-17T11:23:55.718Z
5e67eda71009063689407484
/avatars/6d7688faba888d463584eaa02ad011e3.svg
Mike Lewis
user
mikelewis0
675ffc2a79e2036a7090d50b
Ari Holtzman
admin_assigned
2024-12-17T11:23:40.460Z
6508c81db44445e9b36f3a1e
/avatars/7d4b257398c601b482b9655f046021f3.svg
Ari Holtzman
user
ariholtzman
675ffc2a79e2036a7090d50c
Srinivasan Iyer
Byte Latent Transformer: Patches Scale Better Than Tokens
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM
architecture that, for the first time, matches tokenization-based LLM
performance at scale with significant improvements in inference efficiency and
robustness. BLT encodes bytes into dynamically sized patches, which serve as
the primary units of computation. Patches are segmented based on the entropy of
the next byte, allocating more compute and model capacity where increased data
complexity demands it. We present the first FLOP controlled scaling study of
byte-level models up to 8B parameters and 4T training bytes. Our results
demonstrate the feasibility of scaling models trained on raw bytes without a
fixed vocabulary. Both training and inference efficiency improve due to
dynamically selecting long patches when data is predictable, along with
qualitative improvements on reasoning and long tail generalization. Overall,
for fixed inference costs, BLT shows significantly better scaling than
tokenization-based models, by simultaneously growing both patch and model size.
A Byte Latent Transformer (BLT) matches tokenization-based LLM performance at scale with improved inference efficiency and robustness, using entropy-based byte patching.
Byte Latent Transformer (BLT) byte-level LLM dynamically sized patches entropy FLOP controlled scaling byte-level models model capacity long patches reasoning long tail generalization