title: ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
summary: Large language models (LLMs) have demonstrated impressive capabilities across
  various tasks, but their performance is highly sensitive to the prompts
  utilized. This variability poses challenges for accurate assessment and user
  satisfaction. Current research frequently overlooks instance-level prompt
  variations and their implications on subjective evaluations. To address these
  shortcomings, we introduce ProSA, a framework designed to evaluate and
  comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity
  metric, PromptSensiScore, and leverages decoding confidence to elucidate
  underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers
  that prompt sensitivity fluctuates across datasets and models, with larger
  models exhibiting enhanced robustness. We observe that few-shot examples can
  alleviate this sensitivity issue, and subjective evaluations are also
  susceptible to prompt sensitivities, particularly in complex,
  reasoning-oriented tasks. Furthermore, our findings indicate that higher model
  confidence correlates with increased prompt robustness. We believe this work
  will serve as a helpful tool in studying prompt sensitivity of LLMs. The
  project is released at: https://github.com/open-compass/ProSA .
paper_authors._id: 671076a0eac19807265e1f3f
paper_authors.name: Jingming Zhuo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-17T02:29:54.659Z
paper_authors._id: 6530c7263976e5f4412ba737
paper_authors.avatarUrl: /avatars/8b4d9d847a9e115c3da8cad629bd0a41.svg
paper_authors.fullname: Jingming Zhuo
paper_authors.type: user
paper_authors.user: JingmingZ
paper_authors._id: 671076a0eac19807265e1f40
paper_authors.name: Songyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:43.913Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 671076a0eac19807265e1f41
paper_authors.name: Xinyu Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:02:10.304Z
paper_authors._id: 64f5f8dd9b17cd59c453c57f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
paper_authors.fullname: Xinyu Fang
paper_authors.type: user
paper_authors.user: nebulae09
paper_authors._id: 671076a0eac19807265e1f42
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:41.809Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 671076a0eac19807265e1f43
paper_authors.name: Dahua Lin
paper_authors._id: 671076a0eac19807265e1f44
paper_authors.name: Kai Chen
paper_title: ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
paper_summary: Large language models (LLMs) have demonstrated impressive capabilities across
  various tasks, but their performance is highly sensitive to the prompts
  utilized. This variability poses challenges for accurate assessment and user
  satisfaction. Current research frequently overlooks instance-level prompt
  variations and their implications on subjective evaluations. To address these
  shortcomings, we introduce ProSA, a framework designed to evaluate and
  comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity
  metric, PromptSensiScore, and leverages decoding confidence to elucidate
  underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers
  that prompt sensitivity fluctuates across datasets and models, with larger
  models exhibiting enhanced robustness. We observe that few-shot examples can
  alleviate this sensitivity issue, and subjective evaluations are also
  susceptible to prompt sensitivities, particularly in complex,
  reasoning-oriented tasks. Furthermore, our findings indicate that higher model
  confidence correlates with increased prompt robustness. We believe this work
  will serve as a helpful tool in studying prompt sensitivity of LLMs. The
  project is released at: https://github.com/open-compass/ProSA .
paper_ai_keywords: large language models (LLMs) prompts subjective evaluations ProSA PromptSensiScore decoding confidence prompt sensitivity datasets models few-shot examples model confidence prompt robustness