Uncertainty is Fragile: Manipulating Uncertainty in Large Language
  Models
Large Language Models (LLMs) are employed across various high-stakes domains,
where the reliability of their outputs is crucial. One commonly used method to
assess the reliability of LLMs' responses is uncertainty estimation, which
gauges the likelihood of their answers being correct. While many studies focus
on improving the accuracy of uncertainty estimations for LLMs, our research
investigates the fragility of uncertainty estimation and explores potential
attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which,
when activated by a specific trigger in the input, manipulates the model's
uncertainty without affecting the final output. Specifically, the proposed
backdoor attack method can alter an LLM's output probability distribution,
causing the probability distribution to converge towards an attacker-predefined
distribution while ensuring that the top-1 prediction remains unchanged. Our
experimental results demonstrate that this attack effectively undermines the
model's self-evaluation reliability in multiple-choice questions. For instance,
we achieved a 100 attack success rate (ASR) across three different triggering
strategies in four models. Further, we investigate whether this manipulation
generalizes across different prompts and domains. This work highlights a
significant threat to the reliability of LLMs and underscores the need for
future defenses against such attacks. The code is available at
https://github.com/qcznlp/uncertainty_attack.
6697e90961c9b6bec377d0e9
Qingcheng Zeng
claimed_verified
2025-05-26T08:50:21.100Z
63bc77661374e3ef9135735f
/avatars/94b04545ed9d30bfe58691672a0b5618.svg
Qingcheng Zeng
user
qcz
6697e90961c9b6bec377d0ea
Mingyu Jin
6697e90961c9b6bec377d0eb
Qinkai Yu
6697e90961c9b6bec377d0ec
Zhenting Wang
claimed_verified
2024-12-30T19:39:44.335Z
64dfcc62e8b6f3f3baa950e0
/avatars/21bbff67d46c08044efe2406575aa77e.svg
Zhenting Wang
user
ztwang
6697e90961c9b6bec377d0ed
Wenyue Hua
6697e90961c9b6bec377d0ee
Zihao Zhou
6697e90961c9b6bec377d0ef
Guangyan Sun
6697e90961c9b6bec377d0f0
Yanda Meng
6697e90961c9b6bec377d0f1
Shiqing Ma
6697e90961c9b6bec377d0f2
Qifan Wang
6697e90961c9b6bec377d0f3
Felix Juefei-Xu
claimed_verified
2024-12-16T16:55:45.059Z
6417cf37dce1e4c0229f17b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
Felix Xu
user
katanaxu
6697e90961c9b6bec377d0f4
Kaize Ding
6697e90961c9b6bec377d0f5
Fan Yang
6697e90961c9b6bec377d0f6
Ruixiang Tang
6697e90961c9b6bec377d0f7
Yongfeng Zhang
Uncertainty is Fragile: Manipulating Uncertainty in Large Language
  Models
Large Language Models (LLMs) are employed across various high-stakes domains,
where the reliability of their outputs is crucial. One commonly used method to
assess the reliability of LLMs' responses is uncertainty estimation, which
gauges the likelihood of their answers being correct. While many studies focus
on improving the accuracy of uncertainty estimations for LLMs, our research
investigates the fragility of uncertainty estimation and explores potential
attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which,
when activated by a specific trigger in the input, manipulates the model's
uncertainty without affecting the final output. Specifically, the proposed
backdoor attack method can alter an LLM's output probability distribution,
causing the probability distribution to converge towards an attacker-predefined
distribution while ensuring that the top-1 prediction remains unchanged. Our
experimental results demonstrate that this attack effectively undermines the
model's self-evaluation reliability in multiple-choice questions. For instance,
we achieved a 100 attack success rate (ASR) across three different triggering
strategies in four models. Further, we investigate whether this manipulation
generalizes across different prompts and domains. This work highlights a
significant threat to the reliability of LLMs and underscores the need for
future defenses against such attacks. The code is available at
https://github.com/qcznlp/uncertainty_attack.
Researchers demonstrate that backdoor attacks can manipulate the uncertainty estimation of LLMs without altering their outputs, significantly compromising their reliability.
LLMs uncertainty estimation backdoor trigger output probability distribution multiple-choice questions attack success rate prompts domains