title: CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM
    APIs
summary: As post-training techniques evolve, large language models (LLMs) are
  increasingly augmented with structured multi-step reasoning abilities, often
  optimized through reinforcement learning. These reasoning-enhanced models
  outperform standard LLMs on complex tasks and now underpin many commercial LLM
  APIs. However, to protect proprietary behavior and reduce verbosity, providers
  typically conceal the reasoning traces while returning only the final answer.
  This opacity introduces a critical transparency gap: users are billed for
  invisible reasoning tokens, which often account for the majority of the cost,
  yet have no means to verify their authenticity. This opens the door to token
  count inflation, where providers may overreport token usage or inject
  synthetic, low-effort tokens to inflate charges. To address this issue, we
  propose CoIn, a verification framework that audits both the quantity and
  semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from
  token embedding fingerprints to check token counts, and uses embedding-based
  relevance matching to detect fabricated reasoning content. Experiments
  demonstrate that CoIn, when deployed as a trusted third-party auditor, can
  effectively detect token count inflation with a success rate reaching up to
  94.7%, showing the strong ability to restore billing transparency in opaque LLM
  services. The dataset and code are available at
  https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.
paper_authors._id: 682e0b38d8a198bf9e193d7a
paper_authors.name: Guoheng Sun
paper_authors._id: 682e0b38d8a198bf9e193d7b
paper_authors.name: Ziyao Wang
paper_authors._id: 682e0b38d8a198bf9e193d7c
paper_authors.name: Bowei Tian
paper_authors._id: 682e0b38d8a198bf9e193d7d
paper_authors.name: Meng Liu
paper_authors._id: 682e0b38d8a198bf9e193d7e
paper_authors.name: Zheyu Shen
paper_authors._id: 682e0b38d8a198bf9e193d7f
paper_authors.name: Shwai He
paper_authors._id: 682e0b38d8a198bf9e193d80
paper_authors.name: Yexiao He
paper_authors._id: 682e0b38d8a198bf9e193d81
paper_authors.name: Wanghao Ye
paper_authors._id: 682e0b38d8a198bf9e193d82
paper_authors.name: Yiting Wang
paper_authors._id: 682e0b38d8a198bf9e193d83
paper_authors.name: Ang Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-21T17:20:57.748Z
paper_authors._id: 64a842b1221e6292c9fa7a45
paper_authors.avatarUrl: /avatars/7f67f357c830ed6dade7ed7548ac1f05.svg
paper_authors.fullname: Ang Li
paper_authors.type: user
paper_authors.user: charleslipku
paper_title: CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM
    APIs
paper_summary: As post-training techniques evolve, large language models (LLMs) are
  increasingly augmented with structured multi-step reasoning abilities, often
  optimized through reinforcement learning. These reasoning-enhanced models
  outperform standard LLMs on complex tasks and now underpin many commercial LLM
  APIs. However, to protect proprietary behavior and reduce verbosity, providers
  typically conceal the reasoning traces while returning only the final answer.
  This opacity introduces a critical transparency gap: users are billed for
  invisible reasoning tokens, which often account for the majority of the cost,
  yet have no means to verify their authenticity. This opens the door to token
  count inflation, where providers may overreport token usage or inject
  synthetic, low-effort tokens to inflate charges. To address this issue, we
  propose CoIn, a verification framework that audits both the quantity and
  semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from
  token embedding fingerprints to check token counts, and uses embedding-based
  relevance matching to detect fabricated reasoning content. Experiments
  demonstrate that CoIn, when deployed as a trusted third-party auditor, can
  effectively detect token count inflation with a success rate reaching up to
  94.7%, showing the strong ability to restore billing transparency in opaque LLM
  services. The dataset and code are available at
  https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.
paper_ai_summary: A verification framework named CoIn is proposed to audit and validate the quantity and authenticity of hidden tokens in large language models to ensure billing transparency.
paper_ai_keywords: large language models reinforcement learning structured multi-step reasoning token count inflation verifiable hash tree token embedding fingerprints embedding-based relevance matching