MambaByte: Token-free Selective State Space Model
Token-free language models learn directly from raw bytes and remove the bias
of subword tokenization. Operating on bytes, however, results in significantly
longer sequences, and standard autoregressive Transformers scale poorly in such
settings. We experiment with MambaByte, a token-free adaptation of the Mamba
state space model, trained autoregressively on byte sequences. Our experiments
indicate the computational efficiency of MambaByte compared to other byte-level
models. We also find MambaByte to be competitive with and even outperform
state-of-the-art subword Transformers. Furthermore, owing to linear scaling in
length, MambaByte benefits from fast inference compared to Transformers. Our
findings establish the viability of MambaByte in enabling token-free language
modeling.
65b1c0adf32c79fea71cc3ae
Junxiong Wang
admin_assigned
2024-01-25T08:50:20.900Z
63dc68bea99b2c8a7c20f1d0
/avatars/cf8ddc91415ef1f895803f4390ff1f6f.svg
Junxiong Wang
user
JunxiongWang
65b1c0adf32c79fea71cc3af
Tushaar Gangavarapu
claimed_verified
2024-01-25T08:04:08.230Z
636da52ff18ddf24eaa2e813
https://cdn-avatars.huggingface.co/v1/production/uploads/636da52ff18ddf24eaa2e813/n11v_Hefsv_MeOV3KX2A-.jpeg
Tushaar Gangavarapu
user
tushaar
65b1c0adf32c79fea71cc3b0
Jing Nathan Yan
65b1c0adf32c79fea71cc3b1
Alexander M Rush
claimed_verified
2024-09-20T07:49:09.561Z
5e694f697127cb58dde42fe1
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/J21QWWtFOKqhJTF5qIMlD.jpeg
Sasha Rush
user
srush
MambaByte: Token-free Selective State Space Model
Token-free language models learn directly from raw bytes and remove the bias
of subword tokenization. Operating on bytes, however, results in significantly
longer sequences, and standard autoregressive Transformers scale poorly in such
settings. We experiment with MambaByte, a token-free adaptation of the Mamba
state space model, trained autoregressively on byte sequences. Our experiments
indicate the computational efficiency of MambaByte compared to other byte-level
models. We also find MambaByte to be competitive with and even outperform
state-of-the-art subword Transformers. Furthermore, owing to linear scaling in
length, MambaByte benefits from fast inference compared to Transformers. Our
findings establish the viability of MambaByte in enabling token-free language
modeling.
MambaByte, a token-free byte-level autoregressive model, demonstrates computational efficiency and competitive performance compared to subword token-based models, with the added benefit of fast inference due to linear length scaling.
MambaByte token-free language models subword tokenization autoregressive Transformers byte sequences state space model Mamba state space model