title: Tokenization Falling Short: The Curse of Tokenization
summary: Language models typically tokenize raw text into sequences of subword
  identifiers from a predefined vocabulary, a process inherently sensitive to
  typographical errors, length variations, and largely oblivious to the internal
  structure of tokens-issues we term the curse of tokenization. In this study, we
  delve into these drawbacks and demonstrate that large language models (LLMs)
  remain susceptible to these problems. This study systematically investigates
  these challenges and their impact on LLMs through three critical research
  questions: (1) complex problem solving, (2) token structure probing, and (3)
  resilience to typographical variation. Our findings reveal that scaling model
  parameters can mitigate the issue of tokenization; however, LLMs still suffer
  from biases induced by typos and other text format variations. Our experiments
  show that subword regularization such as BPE-dropout can mitigate this issue.
  We will release our code and data to facilitate further research.
paper_authors._id: 66728498c12108af6ce54bf4
paper_authors.name: Yekun Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:34:25.974Z
paper_authors._id: 626a6ce8adf559c88de9ace0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651142275247-626a6ce8adf559c88de9ace0.jpeg
paper_authors.fullname: CYK
paper_authors.type: user
paper_authors.user: cyk1337
paper_authors._id: 66728498c12108af6ce54bf5
paper_authors.name: Yewei Fang
paper_authors._id: 66728498c12108af6ce54bf6
paper_authors.name: Qiwei Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:08:50.279Z
paper_authors._id: 6346be3a8c68aacc42d8132a
paper_authors.avatarUrl: /avatars/cca3d86318e1046c911cdf82cb93201c.svg
paper_authors.fullname: Qiwei Peng
paper_authors.type: user
paper_authors.user: Punchwe
paper_authors._id: 66728498c12108af6ce54bf7
paper_authors.name: Xuhong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:08:59.989Z
paper_authors._id: 63e9a0d8ca4fc7d30df0a1d6
paper_authors.avatarUrl: /avatars/c996985b6b6055ba76c80331c8f21c7f.svg
paper_authors.fullname: LiXuhong
paper_authors.type: user
paper_authors.user: lixuhong
paper_title: Tokenization Falling Short: The Curse of Tokenization
paper_summary: Language models typically tokenize raw text into sequences of subword
  identifiers from a predefined vocabulary, a process inherently sensitive to
  typographical errors, length variations, and largely oblivious to the internal
  structure of tokens-issues we term the curse of tokenization. In this study, we
  delve into these drawbacks and demonstrate that large language models (LLMs)
  remain susceptible to these problems. This study systematically investigates
  these challenges and their impact on LLMs through three critical research
  questions: (1) complex problem solving, (2) token structure probing, and (3)
  resilience to typographical variation. Our findings reveal that scaling model
  parameters can mitigate the issue of tokenization; however, LLMs still suffer
  from biases induced by typos and other text format variations. Our experiments
  show that subword regularization such as BPE-dropout can mitigate this issue.
  We will release our code and data to facilitate further research.
paper_ai_keywords: tokenization subword identifiers typographical errors length variations token structure probing large language models (LLMs) complex problem solving resilience to typographical variation BPE-dropout