title: Uncertainty is Fragile: Manipulating Uncertainty in Large Language
    Models
summary: Large Language Models (LLMs) are employed across various high-stakes domains,
  where the reliability of their outputs is crucial. One commonly used method to
  assess the reliability of LLMs' responses is uncertainty estimation, which
  gauges the likelihood of their answers being correct. While many studies focus
  on improving the accuracy of uncertainty estimations for LLMs, our research
  investigates the fragility of uncertainty estimation and explores potential
  attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which,
  when activated by a specific trigger in the input, manipulates the model's
  uncertainty without affecting the final output. Specifically, the proposed
  backdoor attack method can alter an LLM's output probability distribution,
  causing the probability distribution to converge towards an attacker-predefined
  distribution while ensuring that the top-1 prediction remains unchanged. Our
  experimental results demonstrate that this attack effectively undermines the
  model's self-evaluation reliability in multiple-choice questions. For instance,
  we achieved a 100 attack success rate (ASR) across three different triggering
  strategies in four models. Further, we investigate whether this manipulation
  generalizes across different prompts and domains. This work highlights a
  significant threat to the reliability of LLMs and underscores the need for
  future defenses against such attacks. The code is available at
  https://github.com/qcznlp/uncertainty_attack.
paper_authors._id: 6697e90961c9b6bec377d0e9
paper_authors.name: Qingcheng Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:50:21.100Z
paper_authors._id: 63bc77661374e3ef9135735f
paper_authors.avatarUrl: /avatars/94b04545ed9d30bfe58691672a0b5618.svg
paper_authors.fullname: Qingcheng Zeng
paper_authors.type: user
paper_authors.user: qcz
paper_authors._id: 6697e90961c9b6bec377d0ea
paper_authors.name: Mingyu Jin
paper_authors._id: 6697e90961c9b6bec377d0eb
paper_authors.name: Qinkai Yu
paper_authors._id: 6697e90961c9b6bec377d0ec
paper_authors.name: Zhenting Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:39:44.335Z
paper_authors._id: 64dfcc62e8b6f3f3baa950e0
paper_authors.avatarUrl: /avatars/21bbff67d46c08044efe2406575aa77e.svg
paper_authors.fullname: Zhenting Wang
paper_authors.type: user
paper_authors.user: ztwang
paper_authors._id: 6697e90961c9b6bec377d0ed
paper_authors.name: Wenyue Hua
paper_authors._id: 6697e90961c9b6bec377d0ee
paper_authors.name: Zihao Zhou
paper_authors._id: 6697e90961c9b6bec377d0ef
paper_authors.name: Guangyan Sun
paper_authors._id: 6697e90961c9b6bec377d0f0
paper_authors.name: Yanda Meng
paper_authors._id: 6697e90961c9b6bec377d0f1
paper_authors.name: Shiqing Ma
paper_authors._id: 6697e90961c9b6bec377d0f2
paper_authors.name: Qifan Wang
paper_authors._id: 6697e90961c9b6bec377d0f3
paper_authors.name: Felix Juefei-Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T16:55:45.059Z
paper_authors._id: 6417cf37dce1e4c0229f17b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
paper_authors.fullname: Felix Xu
paper_authors.type: user
paper_authors.user: katanaxu
paper_authors._id: 6697e90961c9b6bec377d0f4
paper_authors.name: Kaize Ding
paper_authors._id: 6697e90961c9b6bec377d0f5
paper_authors.name: Fan Yang
paper_authors._id: 6697e90961c9b6bec377d0f6
paper_authors.name: Ruixiang Tang
paper_authors._id: 6697e90961c9b6bec377d0f7
paper_authors.name: Yongfeng Zhang
paper_title: Uncertainty is Fragile: Manipulating Uncertainty in Large Language
    Models
paper_summary: Large Language Models (LLMs) are employed across various high-stakes domains,
  where the reliability of their outputs is crucial. One commonly used method to
  assess the reliability of LLMs' responses is uncertainty estimation, which
  gauges the likelihood of their answers being correct. While many studies focus
  on improving the accuracy of uncertainty estimations for LLMs, our research
  investigates the fragility of uncertainty estimation and explores potential
  attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which,
  when activated by a specific trigger in the input, manipulates the model's
  uncertainty without affecting the final output. Specifically, the proposed
  backdoor attack method can alter an LLM's output probability distribution,
  causing the probability distribution to converge towards an attacker-predefined
  distribution while ensuring that the top-1 prediction remains unchanged. Our
  experimental results demonstrate that this attack effectively undermines the
  model's self-evaluation reliability in multiple-choice questions. For instance,
  we achieved a 100 attack success rate (ASR) across three different triggering
  strategies in four models. Further, we investigate whether this manipulation
  generalizes across different prompts and domains. This work highlights a
  significant threat to the reliability of LLMs and underscores the need for
  future defenses against such attacks. The code is available at
  https://github.com/qcznlp/uncertainty_attack.
paper_ai_summary: Researchers demonstrate that backdoor attacks can manipulate the uncertainty estimation of LLMs without altering their outputs, significantly compromising their reliability.
paper_ai_keywords: LLMs uncertainty estimation backdoor trigger output probability distribution multiple-choice questions attack success rate prompts domains