title: DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT
    Models
summary: Generative Pre-trained Transformer (GPT) models have exhibited exciting
  progress in capabilities, capturing the interest of practitioners and the
  public alike. Yet, while the literature on the trustworthiness of GPT models
  remains limited, practitioners have proposed employing capable GPT models for
  sensitive applications to healthcare and finance - where mistakes can be
  costly. To this end, this work proposes a comprehensive trustworthiness
  evaluation for large language models with a focus on GPT-4 and GPT-3.5,
  considering diverse perspectives - including toxicity, stereotype bias,
  adversarial robustness, out-of-distribution robustness, robustness on
  adversarial demonstrations, privacy, machine ethics, and fairness. Based on our
  evaluations, we discover previously unpublished vulnerabilities to
  trustworthiness threats. For instance, we find that GPT models can be easily
  misled to generate toxic and biased outputs and leak private information in
  both training data and conversation history. We also find that although GPT-4
  is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more
  vulnerable given jailbreaking system or user prompts, potentially due to the
  reason that GPT-4 follows the (misleading) instructions more precisely. Our
  work illustrates a comprehensive trustworthiness evaluation of GPT models and
  sheds light on the trustworthiness gaps. Our benchmark is publicly available at
  https://decodingtrust.github.io/.
paper_authors._id: 649250cd48acce7afaff18aa
paper_authors.name: Boxin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T17:53:55.200Z
paper_authors._id: 6493391101196c83393b3104
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6493391101196c83393b3104/C3bl7zYr-91BVxweA2KeW.jpeg
paper_authors.fullname: Boxin Wang
paper_authors.type: user
paper_authors.user: boxin-wbx
paper_authors._id: 649250cd48acce7afaff18ab
paper_authors.name: Weixin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-27T11:06:27.784Z
paper_authors._id: 64ff9065520f8d7b4de6df4f
paper_authors.avatarUrl: /avatars/e8229e4d3fc96e1913cae2f09a034989.svg
paper_authors.fullname: Weixin Chen
paper_authors.type: user
paper_authors.user: weixinchen
paper_authors._id: 649250cd48acce7afaff18ac
paper_authors.name: Hengzhi Pei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T14:37:41.040Z
paper_authors._id: 63082e14b052e6916dbb9d7d
paper_authors.avatarUrl: /avatars/2ca8c5aed9a3094b048fe27e9e9ce71e.svg
paper_authors.fullname: Hengzhi Pei
paper_authors.type: user
paper_authors.user: acphile
paper_authors._id: 649250cd48acce7afaff18ad
paper_authors.name: Chulin Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T14:42:33.435Z
paper_authors._id: 645867e3ead43697df2ca5fb
paper_authors.avatarUrl: /avatars/a0f0e5d975898413be456667c0cbd2a8.svg
paper_authors.fullname: Chulin Xie
paper_authors.type: user
paper_authors.user: alphapav
paper_authors._id: 649250cd48acce7afaff18ae
paper_authors.name: Mintong Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T14:43:00.164Z
paper_authors._id: 63f92d5f1ed25bd00c39c52f
paper_authors.avatarUrl: /avatars/9902fd9b95ed2708a8cc3747933213be.svg
paper_authors.fullname: Mintong Kang
paper_authors.type: user
paper_authors.user: Cometkmt
paper_authors._id: 649250cd48acce7afaff18af
paper_authors.name: Chenhui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T14:44:11.448Z
paper_authors._id: 64522233ea94bf023430dd95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CVDqDeJ_fLTULhCTTSogb.png
paper_authors.fullname: Chenhui Zhang
paper_authors.type: user
paper_authors.user: danielz01
paper_authors._id: 649250cd48acce7afaff18b0
paper_authors.name: Chejian Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T20:46:10.666Z
paper_authors._id: 64931c786b2ee2f5ef96ab25
paper_authors.avatarUrl: /avatars/f7e6d205a88113a609a84ca6d304babb.svg
paper_authors.fullname: Chejian Xu
paper_authors.type: user
paper_authors.user: chejian
paper_authors._id: 649250cd48acce7afaff18b1
paper_authors.name: Zidi Xiong
paper_authors._id: 649250cd48acce7afaff18b2
paper_authors.name: Ritik Dutta
paper_authors._id: 649250cd48acce7afaff18b3
paper_authors.name: Rylan Schaeffer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T14:45:16.910Z
paper_authors._id: 612abe088ef01bc4520e106c
paper_authors.avatarUrl: /avatars/7576cedd550440c22e17f3d7a4d19787.svg
paper_authors.fullname: Rylan Schaeffer
paper_authors.type: user
paper_authors.user: RylanSchaeffer
paper_authors._id: 649250cd48acce7afaff18b4
paper_authors.name: Sang T. Truong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T20:46:23.115Z
paper_authors._id: 6316965493ab42acfb0aa366
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6316965493ab42acfb0aa366/jINtCXxm2LjTWgqJ50EJ_.png
paper_authors.fullname: Sang Truong
paper_authors.type: user
paper_authors.user: sangttruong
paper_authors._id: 649250cd48acce7afaff18b5
paper_authors.name: Simran Arora
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:55:05.262Z
paper_authors._id: 62703f4bbd9c82ff64c2f99f
paper_authors.avatarUrl: /avatars/8ed16c6b38a06fd009c39d26f279a6a9.svg
paper_authors.fullname: Simran
paper_authors.type: user
paper_authors.user: simarora
paper_authors._id: 649250cd48acce7afaff18b6
paper_authors.name: Mantas Mazeika
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T14:46:38.304Z
paper_authors._id: 622bbdeb47e49335b2128a06
paper_authors.avatarUrl: /avatars/acaa6792da1a79a1ba91bbd148c85087.svg
paper_authors.fullname: Mantas Mazeika
paper_authors.type: user
paper_authors.user: mmazeika
paper_authors._id: 649250cd48acce7afaff18b7
paper_authors.name: Dan Hendrycks
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T14:47:20.912Z
paper_authors._id: 63f55aa1b51da4d61da9c96b
paper_authors.avatarUrl: /avatars/56cf9c2d8295c4549248d3b0a4933043.svg
paper_authors.fullname: Dan Hendrycks
paper_authors.type: user
paper_authors.user: hendrycks
paper_authors._id: 649250cd48acce7afaff18b8
paper_authors.name: Zinan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:45:01.450Z
paper_authors._id: 64c832a8c547ed5243d29630
paper_authors.avatarUrl: /avatars/59d1975634e84095b69423c02441d453.svg
paper_authors.fullname: Zinan Lin
paper_authors.type: user
paper_authors.user: fjxmlzn
paper_authors._id: 649250cd48acce7afaff18b9
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T16:26:56.192Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 649250cd48acce7afaff18ba
paper_authors.name: Sanmi Koyejo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T20:46:14.847Z
paper_authors._id: 64931e7e2da595588288f161
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
paper_authors.fullname: Sanmi Koyejo
paper_authors.type: user
paper_authors.user: sanmikoyejo
paper_authors._id: 649250cd48acce7afaff18bb
paper_authors.name: Dawn Song
paper_authors._id: 649250cd48acce7afaff18bc
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T20:46:18.730Z
paper_authors._id: 6493236b70d925ae8050a1bf
paper_authors.avatarUrl: /avatars/b16069de1445cfa8608567175deaa2ae.svg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: BoLi-aisecure
paper_title: DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT
    Models
paper_summary: Generative Pre-trained Transformer (GPT) models have exhibited exciting
  progress in capabilities, capturing the interest of practitioners and the
  public alike. Yet, while the literature on the trustworthiness of GPT models
  remains limited, practitioners have proposed employing capable GPT models for
  sensitive applications to healthcare and finance - where mistakes can be
  costly. To this end, this work proposes a comprehensive trustworthiness
  evaluation for large language models with a focus on GPT-4 and GPT-3.5,
  considering diverse perspectives - including toxicity, stereotype bias,
  adversarial robustness, out-of-distribution robustness, robustness on
  adversarial demonstrations, privacy, machine ethics, and fairness. Based on our
  evaluations, we discover previously unpublished vulnerabilities to
  trustworthiness threats. For instance, we find that GPT models can be easily
  misled to generate toxic and biased outputs and leak private information in
  both training data and conversation history. We also find that although GPT-4
  is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more
  vulnerable given jailbreaking system or user prompts, potentially due to the
  reason that GPT-4 follows the (misleading) instructions more precisely. Our
  work illustrates a comprehensive trustworthiness evaluation of GPT models and
  sheds light on the trustworthiness gaps. Our benchmark is publicly available at
  https://decodingtrust.github.io/.
paper_ai_keywords: generative pre-trained transformer (GPT) models toxicity stereotype bias adversarial robustness out-of-distribution robustness adversarial demonstrations privacy machine ethics fairness trustworthiness threats jailbreaking system user prompts