title: Date Fragments: A Hidden Bottleneck of Tokenization for Temporal
    Reasoning
summary: Modern BPE tokenizers often split calendar dates into meaningless fragments,
  e.g., 20250312 rightarrow 202, 503, 12, inflating token counts and obscuring
  the inherent structure needed for robust temporal reasoning. In this work, we
  (1) introduce a simple yet interpretable metric, termed date fragmentation
  ratio, that measures how faithfully a tokenizer preserves multi-digit date
  components; (2) release DateAugBench, a suite of 6500 examples spanning three
  temporal reasoning tasks: context-based date resolution, format-invariance
  puzzles, and date arithmetic across historical, contemporary, and future
  regimes; and (3) through layer-wise probing and causal attention-hop analyses,
  uncover an emergent date-abstraction mechanism whereby large language models
  stitch together the fragments of month, day, and year components for temporal
  reasoning. Our experiments show that excessive fragmentation correlates with
  accuracy drops of up to 10 points on uncommon dates like historical and
  futuristic dates. Further, we find that the larger the model, the faster the
  emergent date abstraction that heals date fragments is accomplished. Lastly, we
  observe a reasoning path that LLMs follow to assemble date fragments, typically
  differing from human interpretation (year rightarrow month rightarrow
  day).
paper_authors._id: 68302befc518550cc0c2e505
paper_authors.name: Gagan Bhatia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:14:47.929Z
paper_authors._id: 60394599033b61166496163b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1614366097007-noauth.jpeg
paper_authors.fullname: Gagan Bhatia
paper_authors.type: user
paper_authors.user: gagan3012
paper_authors._id: 68302befc518550cc0c2e506
paper_authors.name: Maxime Peyrard
paper_authors._id: 68302befc518550cc0c2e507
paper_authors.name: Wei Zhao
paper_title: Date Fragments: A Hidden Bottleneck of Tokenization for Temporal
    Reasoning
paper_summary: Modern BPE tokenizers often split calendar dates into meaningless fragments,
  e.g., 20250312 rightarrow 202, 503, 12, inflating token counts and obscuring
  the inherent structure needed for robust temporal reasoning. In this work, we
  (1) introduce a simple yet interpretable metric, termed date fragmentation
  ratio, that measures how faithfully a tokenizer preserves multi-digit date
  components; (2) release DateAugBench, a suite of 6500 examples spanning three
  temporal reasoning tasks: context-based date resolution, format-invariance
  puzzles, and date arithmetic across historical, contemporary, and future
  regimes; and (3) through layer-wise probing and causal attention-hop analyses,
  uncover an emergent date-abstraction mechanism whereby large language models
  stitch together the fragments of month, day, and year components for temporal
  reasoning. Our experiments show that excessive fragmentation correlates with
  accuracy drops of up to 10 points on uncommon dates like historical and
  futuristic dates. Further, we find that the larger the model, the faster the
  emergent date abstraction that heals date fragments is accomplished. Lastly, we
  observe a reasoning path that LLMs follow to assemble date fragments, typically
  differing from human interpretation (year rightarrow month rightarrow
  day).
paper_ai_summary: New DateAugBench benchmarks reveal how modern tokenizers fragment dates, impacting the accuracy of temporal reasoning in large language models, which compensate for fragmentation more effectively as they grow larger.
paper_ai_keywords: BPE tokenizers date fragmentation ratio DateAugBench temporal reasoning tasks context-based date resolution format-invariance puzzles date arithmetic layer-wise probing causal attention-hop analyses date-abstraction mechanism large language models historical dates futuristic dates