title: FoNE: Precise Single-Token Number Embeddings via Fourier Features
summary: Large Language Models (LLMs) typically represent numbers using multiple
  tokens, which requires the model to aggregate these tokens to interpret
  numerical values. This fragmentation makes both training and inference less
  efficient and adversely affects the model's performance on number-related
  tasks. Inspired by the observation that pre-trained LLMs internally learn
  Fourier-like features for number tokens, we propose Fourier Number Embedding
  (FoNE), a novel method that directly maps numbers into the embedding space with
  their Fourier features. FoNE encodes each number as a single token with only
  two embedding dimensions per digit, effectively capturing numerical values
  without fragmentation. This compact representation accelerates both training
  and inference. Compared to traditional subword and digit-wise embeddings, FoNE
  not only reduces computational overhead but also achieves higher accuracy
  across various numerical tasks including addition, subtraction and
  multiplication. On 6-digit decimal addition, FoNE requires 64times less data
  to achieve 99% accuracy than subword and digit-wise embeddings while using
  3times and 6times fewer tokens per number, respectively. Furthermore,
  FoNE is the only method that yields 100% accuracy on over 100,000 test examples
  for addition, subtraction, and multiplication. The codes and visualization are
  available at https://fouriernumber.github.io/.
paper_authors._id: 67b2b58f9edebc815a2a938c
paper_authors.name: Tianyi Zhou
paper_authors._id: 67b2b58f9edebc815a2a938d
paper_authors.name: Deqing Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:32:57.319Z
paper_authors._id: 63c8454e46421a2efe82709d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/3BcSk4KOwAgWHEPVtsAV3.png
paper_authors.fullname: Deqing Fu
paper_authors.type: user
paper_authors.user: deqing
paper_authors._id: 67b2b58f9edebc815a2a938e
paper_authors.name: Mahdi Soltanolkotabi
paper_authors._id: 67b2b58f9edebc815a2a938f
paper_authors.name: Robin Jia
paper_authors._id: 67b2b58f9edebc815a2a9390
paper_authors.name: Vatsal Sharan
paper_title: FoNE: Precise Single-Token Number Embeddings via Fourier Features
paper_summary: Large Language Models (LLMs) typically represent numbers using multiple
  tokens, which requires the model to aggregate these tokens to interpret
  numerical values. This fragmentation makes both training and inference less
  efficient and adversely affects the model's performance on number-related
  tasks. Inspired by the observation that pre-trained LLMs internally learn
  Fourier-like features for number tokens, we propose Fourier Number Embedding
  (FoNE), a novel method that directly maps numbers into the embedding space with
  their Fourier features. FoNE encodes each number as a single token with only
  two embedding dimensions per digit, effectively capturing numerical values
  without fragmentation. This compact representation accelerates both training
  and inference. Compared to traditional subword and digit-wise embeddings, FoNE
  not only reduces computational overhead but also achieves higher accuracy
  across various numerical tasks including addition, subtraction and
  multiplication. On 6-digit decimal addition, FoNE requires 64times less data
  to achieve 99% accuracy than subword and digit-wise embeddings while using
  3times and 6times fewer tokens per number, respectively. Furthermore,
  FoNE is the only method that yields 100% accuracy on over 100,000 test examples
  for addition, subtraction, and multiplication. The codes and visualization are
  available at https://fouriernumber.github.io/.
paper_ai_summary: Fourier Number Embedding (FoNE) improves numerical task performance and efficiency in large language models by directly mapping numbers to Fourier features in a compact embedding space.
paper_ai_keywords: Fourier Number Embedding Fourier features embedding space numerical tasks addition subtraction multiplication subword embeddings digit-wise embeddings