title: Are You Getting What You Pay For? Auditing Model Substitution in LLM
    APIs
summary: The proliferation of Large Language Models (LLMs) accessed via black-box APIs
  introduces a significant trust challenge: users pay for services based on
  advertised model capabilities (e.g., size, performance), but providers may
  covertly substitute the specified model with a cheaper, lower-quality
  alternative to reduce operational costs. This lack of transparency undermines
  fairness, erodes trust, and complicates reliable benchmarking. Detecting such
  substitutions is difficult due to the black-box nature, typically limiting
  interaction to input-output queries. This paper formalizes the problem of model
  substitution detection in LLM APIs. We systematically evaluate existing
  verification techniques, including output-based statistical tests, benchmark
  evaluations, and log probability analysis, under various realistic attack
  scenarios like model quantization, randomized substitution, and benchmark
  evasion. Our findings reveal the limitations of methods relying solely on text
  outputs, especially against subtle or adaptive attacks. While log probability
  analysis offers stronger guarantees when available, its accessibility is often
  limited. We conclude by discussing the potential of hardware-based solutions
  like Trusted Execution Environments (TEEs) as a pathway towards provable model
  integrity, highlighting the trade-offs between security, performance, and
  provider adoption. Code is available at
  https://github.com/sunblaze-ucb/llm-api-audit
paper_authors._id: 67f4a59bf51362f606eb84e8
paper_authors.name: Will Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T16:22:39.251Z
paper_authors._id: 671f7193315a5b40773b36c3
paper_authors.avatarUrl: /avatars/e097da0032341958c57237e4006587ab.svg
paper_authors.fullname: Will Cai
paper_authors.type: user
paper_authors.user: wicai24
paper_authors._id: 67f4a59bf51362f606eb84e9
paper_authors.name: Tianneng Shi
paper_authors._id: 67f4a59bf51362f606eb84ea
paper_authors.name: Xuandong Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T10:36:45.588Z
paper_authors._id: 6275a465597c70eb8949fce5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6275a465597c70eb8949fce5/ph4UogqMurMB0hSXZC38w.png
paper_authors.fullname: Xuandong Zhao
paper_authors.type: user
paper_authors.user: Xuandong
paper_authors._id: 67f4a59bf51362f606eb84eb
paper_authors.name: Dawn Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T10:36:39.333Z
paper_authors._id: 6513179af60393414afd4140
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GAf-FulRRNF-X_Svew1Xj.jpeg
paper_authors.fullname: Xiaowei Song
paper_authors.type: user
paper_authors.user: dawnsong
paper_title: Are You Getting What You Pay For? Auditing Model Substitution in LLM
    APIs
paper_summary: The proliferation of Large Language Models (LLMs) accessed via black-box APIs
  introduces a significant trust challenge: users pay for services based on
  advertised model capabilities (e.g., size, performance), but providers may
  covertly substitute the specified model with a cheaper, lower-quality
  alternative to reduce operational costs. This lack of transparency undermines
  fairness, erodes trust, and complicates reliable benchmarking. Detecting such
  substitutions is difficult due to the black-box nature, typically limiting
  interaction to input-output queries. This paper formalizes the problem of model
  substitution detection in LLM APIs. We systematically evaluate existing
  verification techniques, including output-based statistical tests, benchmark
  evaluations, and log probability analysis, under various realistic attack
  scenarios like model quantization, randomized substitution, and benchmark
  evasion. Our findings reveal the limitations of methods relying solely on text
  outputs, especially against subtle or adaptive attacks. While log probability
  analysis offers stronger guarantees when available, its accessibility is often
  limited. We conclude by discussing the potential of hardware-based solutions
  like Trusted Execution Environments (TEEs) as a pathway towards provable model
  integrity, highlighting the trade-offs between security, performance, and
  provider adoption. Code is available at
  https://github.com/sunblaze-ucb/llm-api-audit
paper_ai_summary: The paper evaluates existing techniques for detecting model substitutions in LLM APIs, highlighting the limitations of output-based methods and the potential of hardware-based solutions.
paper_ai_keywords: Large Language Models (LLMs) black-box APIs model substitution detection verification techniques output-based statistical tests benchmark evaluations log probability analysis model quantization randomized substitution benchmark evasion Trusted Execution Environments (TEEs)