Tokenization Falling Short: The Curse of Tokenization
Language models typically tokenize raw text into sequences of subword
identifiers from a predefined vocabulary, a process inherently sensitive to
typographical errors, length variations, and largely oblivious to the internal
structure of tokens-issues we term the curse of tokenization. In this study, we
delve into these drawbacks and demonstrate that large language models (LLMs)
remain susceptible to these problems. This study systematically investigates
these challenges and their impact on LLMs through three critical research
questions: (1) complex problem solving, (2) token structure probing, and (3)
resilience to typographical variation. Our findings reveal that scaling model
parameters can mitigate the issue of tokenization; however, LLMs still suffer
from biases induced by typos and other text format variations. Our experiments
show that subword regularization such as BPE-dropout can mitigate this issue.
We will release our code and data to facilitate further research.
66728498c12108af6ce54bf4
Yekun Chai
claimed_verified
2024-06-20T07:34:25.974Z
626a6ce8adf559c88de9ace0
https://cdn-avatars.huggingface.co/v1/production/uploads/1651142275247-626a6ce8adf559c88de9ace0.jpeg
CYK
user
cyk1337
66728498c12108af6ce54bf5
Yewei Fang
66728498c12108af6ce54bf6
Qiwei Peng
admin_assigned
2024-06-19T21:08:50.279Z
6346be3a8c68aacc42d8132a
/avatars/cca3d86318e1046c911cdf82cb93201c.svg
Qiwei Peng
user
Punchwe
66728498c12108af6ce54bf7
Xuhong Li
admin_assigned
2024-06-19T21:08:59.989Z
63e9a0d8ca4fc7d30df0a1d6
/avatars/c996985b6b6055ba76c80331c8f21c7f.svg
LiXuhong
user
lixuhong
Tokenization Falling Short: The Curse of Tokenization
Language models typically tokenize raw text into sequences of subword
identifiers from a predefined vocabulary, a process inherently sensitive to
typographical errors, length variations, and largely oblivious to the internal
structure of tokens-issues we term the curse of tokenization. In this study, we
delve into these drawbacks and demonstrate that large language models (LLMs)
remain susceptible to these problems. This study systematically investigates
these challenges and their impact on LLMs through three critical research
questions: (1) complex problem solving, (2) token structure probing, and (3)
resilience to typographical variation. Our findings reveal that scaling model
parameters can mitigate the issue of tokenization; however, LLMs still suffer
from biases induced by typos and other text format variations. Our experiments
show that subword regularization such as BPE-dropout can mitigate this issue.
We will release our code and data to facilitate further research.
Large language models are sensitive to tokenization issues, including typographical errors and subword structure, but parameter scaling and subword regularization can mitigate these problems.
tokenization subword identifiers large language models LLMs token structure resilience typographical variation BPE-dropout