CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM
  APIs
As post-training techniques evolve, large language models (LLMs) are
increasingly augmented with structured multi-step reasoning abilities, often
optimized through reinforcement learning. These reasoning-enhanced models
outperform standard LLMs on complex tasks and now underpin many commercial LLM
APIs. However, to protect proprietary behavior and reduce verbosity, providers
typically conceal the reasoning traces while returning only the final answer.
This opacity introduces a critical transparency gap: users are billed for
invisible reasoning tokens, which often account for the majority of the cost,
yet have no means to verify their authenticity. This opens the door to token
count inflation, where providers may overreport token usage or inject
synthetic, low-effort tokens to inflate charges. To address this issue, we
propose CoIn, a verification framework that audits both the quantity and
semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from
token embedding fingerprints to check token counts, and uses embedding-based
relevance matching to detect fabricated reasoning content. Experiments
demonstrate that CoIn, when deployed as a trusted third-party auditor, can
effectively detect token count inflation with a success rate reaching up to
94.7%, showing the strong ability to restore billing transparency in opaque LLM
services. The dataset and code are available at
https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.
682e0b38d8a198bf9e193d7a
Guoheng Sun
extracted_pending
2025-06-22T00:59:32.072Z
68574f7b063b98493aab2882
/avatars/b146a2f79588e5d579be70b9309b975d.svg
Guoheng Sun
user
ghsun
682e0b38d8a198bf9e193d7b
Ziyao Wang
682e0b38d8a198bf9e193d7c
Bowei Tian
682e0b38d8a198bf9e193d7d
Meng Liu
682e0b38d8a198bf9e193d7e
Zheyu Shen
682e0b38d8a198bf9e193d7f
Shwai He
682e0b38d8a198bf9e193d80
Yexiao He
682e0b38d8a198bf9e193d81
Wanghao Ye
claimed_verified
2025-06-24T08:11:29.476Z
670fe9d4c981c62925b164ac
/avatars/e5f045e885094aa1997aa31642df814a.svg
Wanghao Ye
user
wanghaoy
682e0b38d8a198bf9e193d82
Yiting Wang
682e0b38d8a198bf9e193d83
Ang Li
extracted_confirmed
2025-05-21T17:20:57.748Z
64a842b1221e6292c9fa7a45
/avatars/7f67f357c830ed6dade7ed7548ac1f05.svg
Ang Li
user
charleslipku
CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM
  APIs
As post-training techniques evolve, large language models (LLMs) are
increasingly augmented with structured multi-step reasoning abilities, often
optimized through reinforcement learning. These reasoning-enhanced models
outperform standard LLMs on complex tasks and now underpin many commercial LLM
APIs. However, to protect proprietary behavior and reduce verbosity, providers
typically conceal the reasoning traces while returning only the final answer.
This opacity introduces a critical transparency gap: users are billed for
invisible reasoning tokens, which often account for the majority of the cost,
yet have no means to verify their authenticity. This opens the door to token
count inflation, where providers may overreport token usage or inject
synthetic, low-effort tokens to inflate charges. To address this issue, we
propose CoIn, a verification framework that audits both the quantity and
semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from
token embedding fingerprints to check token counts, and uses embedding-based
relevance matching to detect fabricated reasoning content. Experiments
demonstrate that CoIn, when deployed as a trusted third-party auditor, can
effectively detect token count inflation with a success rate reaching up to
94.7%, showing the strong ability to restore billing transparency in opaque LLM
services. The dataset and code are available at
https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.
A verification framework named CoIn is proposed to audit and validate the quantity and authenticity of hidden tokens in large language models to ensure billing transparency.
large language models reinforcement learning structured multi-step reasoning token count inflation verifiable hash tree token embedding fingerprints embedding-based relevance matching