title: MambaByte: Token-free Selective State Space Model
summary: Token-free language models learn directly from raw bytes and remove the bias
  of subword tokenization. Operating on bytes, however, results in significantly
  longer sequences, and standard autoregressive Transformers scale poorly in such
  settings. We experiment with MambaByte, a token-free adaptation of the Mamba
  state space model, trained autoregressively on byte sequences. Our experiments
  indicate the computational efficiency of MambaByte compared to other byte-level
  models. We also find MambaByte to be competitive with and even outperform
  state-of-the-art subword Transformers. Furthermore, owing to linear scaling in
  length, MambaByte benefits from fast inference compared to Transformers. Our
  findings establish the viability of MambaByte in enabling token-free language
  modeling.
paper_authors._id: 65b1c0adf32c79fea71cc3ae
paper_authors.name: Junxiong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T08:50:20.900Z
paper_authors._id: 63dc68bea99b2c8a7c20f1d0
paper_authors.avatarUrl: /avatars/cf8ddc91415ef1f895803f4390ff1f6f.svg
paper_authors.fullname: Junxiong Wang
paper_authors.type: user
paper_authors.user: JunxiongWang
paper_authors._id: 65b1c0adf32c79fea71cc3af
paper_authors.name: Tushaar Gangavarapu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-25T08:04:08.230Z
paper_authors._id: 636da52ff18ddf24eaa2e813
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636da52ff18ddf24eaa2e813/n11v_Hefsv_MeOV3KX2A-.jpeg
paper_authors.fullname: Tushaar Gangavarapu
paper_authors.type: user
paper_authors.user: tushaar
paper_authors._id: 65b1c0adf32c79fea71cc3b0
paper_authors.name: Jing Nathan Yan
paper_authors._id: 65b1c0adf32c79fea71cc3b1
paper_authors.name: Alexander M Rush
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:49:09.561Z
paper_authors._id: 5e694f697127cb58dde42fe1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/J21QWWtFOKqhJTF5qIMlD.jpeg
paper_authors.fullname: Sasha Rush
paper_authors.type: user
paper_authors.user: srush
paper_title: MambaByte: Token-free Selective State Space Model
paper_summary: Token-free language models learn directly from raw bytes and remove the bias
  of subword tokenization. Operating on bytes, however, results in significantly
  longer sequences, and standard autoregressive Transformers scale poorly in such
  settings. We experiment with MambaByte, a token-free adaptation of the Mamba
  state space model, trained autoregressively on byte sequences. Our experiments
  indicate the computational efficiency of MambaByte compared to other byte-level
  models. We also find MambaByte to be competitive with and even outperform
  state-of-the-art subword Transformers. Furthermore, owing to linear scaling in
  length, MambaByte benefits from fast inference compared to Transformers. Our
  findings establish the viability of MambaByte in enabling token-free language
  modeling.
paper_ai_keywords: token-free language models subword tokenization bytes sequences standard autoregressive Transformers MambaByte token-free adaptation Mamba state space model byte sequences computational efficiency representation engineering (RepE) state-of-the-art subword Transformers linear scaling fast inference token-free language modeling