DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT
  Models
Generative Pre-trained Transformer (GPT) models have exhibited exciting
progress in capabilities, capturing the interest of practitioners and the
public alike. Yet, while the literature on the trustworthiness of GPT models
remains limited, practitioners have proposed employing capable GPT models for
sensitive applications to healthcare and finance - where mistakes can be
costly. To this end, this work proposes a comprehensive trustworthiness
evaluation for large language models with a focus on GPT-4 and GPT-3.5,
considering diverse perspectives - including toxicity, stereotype bias,
adversarial robustness, out-of-distribution robustness, robustness on
adversarial demonstrations, privacy, machine ethics, and fairness. Based on our
evaluations, we discover previously unpublished vulnerabilities to
trustworthiness threats. For instance, we find that GPT models can be easily
misled to generate toxic and biased outputs and leak private information in
both training data and conversation history. We also find that although GPT-4
is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more
vulnerable given jailbreaking system or user prompts, potentially due to the
reason that GPT-4 follows the (misleading) instructions more precisely. Our
work illustrates a comprehensive trustworthiness evaluation of GPT models and
sheds light on the trustworthiness gaps. Our benchmark is publicly available at
https://decodingtrust.github.io/.
649250cd48acce7afaff18aa
Boxin Wang
claimed_verified
2023-06-21T17:53:55.200Z
6493391101196c83393b3104
https://cdn-avatars.huggingface.co/v1/production/uploads/6493391101196c83393b3104/C3bl7zYr-91BVxweA2KeW.jpeg
Boxin Wang
user
boxin-wbx
649250cd48acce7afaff18ab
Weixin Chen
claimed_verified
2023-12-27T11:06:27.784Z
64ff9065520f8d7b4de6df4f
/avatars/e8229e4d3fc96e1913cae2f09a034989.svg
Weixin Chen
user
weixinchen
649250cd48acce7afaff18ac
Hengzhi Pei
admin_assigned
2023-06-21T14:37:41.040Z
63082e14b052e6916dbb9d7d
/avatars/2ca8c5aed9a3094b048fe27e9e9ce71e.svg
Hengzhi Pei
user
acphile
649250cd48acce7afaff18ad
Chulin Xie
admin_assigned
2023-06-21T14:42:33.435Z
645867e3ead43697df2ca5fb
/avatars/a0f0e5d975898413be456667c0cbd2a8.svg
Chulin Xie
user
alphapav
649250cd48acce7afaff18ae
Mintong Kang
admin_assigned
2023-06-21T14:43:00.164Z
63f92d5f1ed25bd00c39c52f
/avatars/9902fd9b95ed2708a8cc3747933213be.svg
Mintong Kang
user
Cometkmt
649250cd48acce7afaff18af
Chenhui Zhang
admin_assigned
2023-06-21T14:44:11.448Z
64522233ea94bf023430dd95
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CVDqDeJ_fLTULhCTTSogb.png
Chenhui Zhang
user
danielz01
649250cd48acce7afaff18b0
Chejian Xu
claimed_verified
2023-06-21T20:46:10.666Z
64931c786b2ee2f5ef96ab25
/avatars/f7e6d205a88113a609a84ca6d304babb.svg
Chejian Xu
user
chejian
649250cd48acce7afaff18b1
Zidi Xiong
649250cd48acce7afaff18b2
Ritik Dutta
649250cd48acce7afaff18b3
Rylan Schaeffer
admin_assigned
2023-06-21T14:45:16.910Z
612abe088ef01bc4520e106c
/avatars/7576cedd550440c22e17f3d7a4d19787.svg
Rylan Schaeffer
user
RylanSchaeffer
649250cd48acce7afaff18b4
Sang T. Truong
claimed_verified
2023-06-21T20:46:23.115Z
6316965493ab42acfb0aa366
https://cdn-avatars.huggingface.co/v1/production/uploads/6316965493ab42acfb0aa366/jINtCXxm2LjTWgqJ50EJ_.png
Sang Truong
user
sangttruong
649250cd48acce7afaff18b5
Simran Arora
claimed_verified
2024-10-21T07:55:05.262Z
62703f4bbd9c82ff64c2f99f
/avatars/8ed16c6b38a06fd009c39d26f279a6a9.svg
Simran
user
simarora
649250cd48acce7afaff18b6
Mantas Mazeika
admin_assigned
2023-06-21T14:46:38.304Z
622bbdeb47e49335b2128a06
/avatars/acaa6792da1a79a1ba91bbd148c85087.svg
Mantas Mazeika
user
mmazeika
649250cd48acce7afaff18b7
Dan Hendrycks
admin_assigned
2023-06-21T14:47:20.912Z
63f55aa1b51da4d61da9c96b
/avatars/56cf9c2d8295c4549248d3b0a4933043.svg
Dan Hendrycks
user
hendrycks
649250cd48acce7afaff18b8
Zinan Lin
claimed_verified
2023-08-01T07:45:01.450Z
64c832a8c547ed5243d29630
https://cdn-avatars.huggingface.co/v1/production/uploads/64c832a8c547ed5243d29630/a46V0xOVRVknM9mUjKkTf.jpeg
Zinan Lin
user
fjxmlzn
649250cd48acce7afaff18b9
Yu Cheng
claimed_verified
2024-10-09T16:26:56.192Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
649250cd48acce7afaff18ba
Sanmi Koyejo
claimed_verified
2023-06-21T20:46:14.847Z
64931e7e2da595588288f161
https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
Sanmi Koyejo
user
sanmikoyejo
649250cd48acce7afaff18bb
Dawn Song
649250cd48acce7afaff18bc
Bo Li
claimed_verified
2023-06-21T20:46:18.730Z
6493236b70d925ae8050a1bf
/avatars/b16069de1445cfa8608567175deaa2ae.svg
Bo Li
user
BoLi-aisecure
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT
  Models
Generative Pre-trained Transformer (GPT) models have exhibited exciting
progress in capabilities, capturing the interest of practitioners and the
public alike. Yet, while the literature on the trustworthiness of GPT models
remains limited, practitioners have proposed employing capable GPT models for
sensitive applications to healthcare and finance - where mistakes can be
costly. To this end, this work proposes a comprehensive trustworthiness
evaluation for large language models with a focus on GPT-4 and GPT-3.5,
considering diverse perspectives - including toxicity, stereotype bias,
adversarial robustness, out-of-distribution robustness, robustness on
adversarial demonstrations, privacy, machine ethics, and fairness. Based on our
evaluations, we discover previously unpublished vulnerabilities to
trustworthiness threats. For instance, we find that GPT models can be easily
misled to generate toxic and biased outputs and leak private information in
both training data and conversation history. We also find that although GPT-4
is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more
vulnerable given jailbreaking system or user prompts, potentially due to the
reason that GPT-4 follows the (misleading) instructions more precisely. Our
work illustrates a comprehensive trustworthiness evaluation of GPT models and
sheds light on the trustworthiness gaps. Our benchmark is publicly available at
https://decodingtrust.github.io/.
A comprehensive trustworthiness evaluation of GPT models, focusing on vulnerabilities related to toxicity, bias, adversarial robustness, privacy, ethics, and fairness, reveals that GPT-4, while generally more trustworthy, can be misled by misleading instructions.
trustedworthiness evaluation large language models GPT-4 GPT-3.5 toxicity stereotype bias adversarial robustness out-of-distribution robustness privacy machine ethics fairness jailbreaking system prompts user prompts