Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic
  Reasoning Limits
Tokenization is the first - and often underappreciated - layer of computation
in language models. While Chain-of-Thought (CoT) prompting enables transformer
models to approximate recurrent computation by externalizing intermediate
steps, we show that the success of such reasoning is fundamentally bounded by
the structure of tokenized inputs. This work presents a theoretical and
empirical investigation into how tokenization schemes, particularly
subword-based methods like byte-pair encoding (BPE), impede symbolic
computation by merging or obscuring atomic reasoning units. We introduce the
notion of Token Awareness to formalize how poor token granularity disrupts
logical alignment and prevents models from generalizing symbolic procedures.
Through systematic evaluation on arithmetic and symbolic tasks, we demonstrate
that token structure dramatically affect reasoning performance, causing failure
even with CoT, while atomically-aligned formats unlock strong generalization,
allowing small models (e.g., GPT-4o-mini) to outperform larger systems (e.g.,
o1) in structured reasoning. Our findings reveal that symbolic reasoning
ability in LLMs is not purely architectural, but deeply conditioned on
token-level representations.
682d388c57686b8c44ede70b
Xiang Zhang
extracted_pending
2025-05-21T02:21:00.902Z
656553d89bf6665f10e3a92d
/avatars/f55b09e249c48ef3fe484afa33a182ae.svg
xiang wyatt zhang
user
Wyattz23
682d388c57686b8c44ede70c
Juntai Cao
682d388c57686b8c44ede70d
Jiaqi Wei
682d388c57686b8c44ede70e
Yiwei Xu
682d388c57686b8c44ede70f
Chenyu You
extracted_confirmed
2025-05-21T02:30:12.849Z
6466d463060756d2854ab3e1
/avatars/4401387180c16472a6823f78aaa86d54.svg
Chenyu You
user
Charlesyooo
Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic
  Reasoning Limits
Tokenization is the first - and often underappreciated - layer of computation
in language models. While Chain-of-Thought (CoT) prompting enables transformer
models to approximate recurrent computation by externalizing intermediate
steps, we show that the success of such reasoning is fundamentally bounded by
the structure of tokenized inputs. This work presents a theoretical and
empirical investigation into how tokenization schemes, particularly
subword-based methods like byte-pair encoding (BPE), impede symbolic
computation by merging or obscuring atomic reasoning units. We introduce the
notion of Token Awareness to formalize how poor token granularity disrupts
logical alignment and prevents models from generalizing symbolic procedures.
Through systematic evaluation on arithmetic and symbolic tasks, we demonstrate
that token structure dramatically affect reasoning performance, causing failure
even with CoT, while atomically-aligned formats unlock strong generalization,
allowing small models (e.g., GPT-4o-mini) to outperform larger systems (e.g.,
o1) in structured reasoning. Our findings reveal that symbolic reasoning
ability in LLMs is not purely architectural, but deeply conditioned on
token-level representations.
Tokenization structure significantly impacts symbolic reasoning in language models by affecting token granularity and alignment, influencing performance even with Chain-of-Thought prompting.
tokenization Chain-of-Thought (CoT) transformer models subword-based methods byte-pair encoding (BPE) Token Awareness symbolic computation logical alignment structured reasoning large language models (LLMs)