TrustLLM: Trustworthiness in Large Language Models
Large language models (LLMs), exemplified by ChatGPT, have gained
considerable attention for their excellent natural language processing
capabilities. Nonetheless, these LLMs present many challenges, particularly in
the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs
emerges as an important topic. This paper introduces TrustLLM, a comprehensive
study of trustworthiness in LLMs, including principles for different dimensions
of trustworthiness, established benchmark, evaluation, and analysis of
trustworthiness for mainstream LLMs, and discussion of open challenges and
future directions. Specifically, we first propose a set of principles for
trustworthy LLMs that span eight different dimensions. Based on these
principles, we further establish a benchmark across six dimensions including
truthfulness, safety, fairness, robustness, privacy, and machine ethics. We
then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of
over 30 datasets. Our findings firstly show that in general trustworthiness and
utility (i.e., functional effectiveness) are positively related. Secondly, our
observations reveal that proprietary LLMs generally outperform most open-source
counterparts in terms of trustworthiness, raising concerns about the potential
risks of widely accessible open-source LLMs. However, a few open-source LLMs
come very close to proprietary ones. Thirdly, it is important to note that some
LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent
that they compromise their utility by mistakenly treating benign prompts as
harmful and consequently not responding. Finally, we emphasize the importance
of ensuring transparency not only in the models themselves but also in the
technologies that underpin trustworthiness. Knowing the specific trustworthy
technologies that have been employed is crucial for analyzing their
effectiveness.
65a09d6ce969415381f1a31d
Lichao Sun
claimed_verified
2024-03-21T08:29:16.487Z
65a52766215aabac489e3468
/avatars/fe05e22cd7e12e961296426434e17c76.svg
Lichao Sun
user
sunlichao137
65a09d6ce969415381f1a31e
Yue Huang
claimed_verified
2025-02-19T09:08:53.072Z
639d94ab7145123e0d44e48a
/avatars/5bb6a65b306d1383c4a8bcd9334b470a.svg
Yue Huang
user
HowieHwong
65a09d6ce969415381f1a31f
Haoran Wang
claimed_verified
2024-01-15T07:42:05.891Z
64b82c659ebb69a79f0073f6
https://cdn-avatars.huggingface.co/v1/production/uploads/64b82c659ebb69a79f0073f6/INEHG7kijEHOhFZMjdBIM.png
Haoran Wang
user
wang2226
65a09d6ce969415381f1a320
Siyuan Wu
65a09d6ce969415381f1a321
Qihui Zhang
claimed_verified
2025-03-21T11:51:37.824Z
67a99d1fef1439e285c4cbec
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VrwUmrY2wsg4sVSIMc--K.png
Qihui Zhang
user
77Hui
65a09d6ce969415381f1a322
Chujie Gao
claimed_verified
2025-02-21T10:01:10.373Z
65a13cb1c5770b27aef2a2bc
/avatars/88ec5b988f10ad9fd4d469ae2fa34680.svg
Chujie Gao
user
Flossie
65a09d6ce969415381f1a323
Yixin Huang
65a09d6ce969415381f1a324
Wenhan Lyu
65a09d6ce969415381f1a325
Yixuan Zhang
65a09d6ce969415381f1a326
Xiner Li
admin_assigned
2024-01-12T10:49:50.423Z
651dc3862c5d36450919d0dc
/avatars/03fb549171d3b103914aa64103e5739e.svg
Xiner Li
user
hyanan16
65a09d6ce969415381f1a327
Zhengliang Liu
65a09d6ce969415381f1a328
Yixin Liu
admin_assigned
2024-01-12T10:50:25.731Z
6244de1c1c560fb11edfca44
/avatars/36558928bd04be7f49837d4c603681d7.svg
Yixin Liu
user
henryL7
65a09d6ce969415381f1a329
Yijue Wang
65a09d6ce969415381f1a32a
Zhikun Zhang
65a09d6ce969415381f1a32b
Bhavya Kailkhura
claimed_verified
2024-05-28T12:44:31.015Z
65cb79db6427380bc21261e2
/avatars/a003eb5d0955417329c1a4170ae65879.svg
Bhavya Kailkhura
user
bhavyakailkhura
65a09d6ce969415381f1a32c
Caiming Xiong
admin_assigned
2024-01-12T10:51:13.398Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
65a09d6ce969415381f1a32d
Chao Zhang
65a09d6ce969415381f1a32e
Chaowei Xiao
65a09d6ce969415381f1a32f
Chunyuan Li
admin_assigned
2024-01-12T10:51:35.436Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
65a09d6ce969415381f1a330
Eric Xing
admin_assigned
2024-01-12T10:51:42.959Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
65a09d6ce969415381f1a331
Furong Huang
admin_assigned
2024-01-12T10:51:53.877Z
64cbc3e2a257a3212c00a115
/avatars/836e61be4aeda2080ddf2db9f2626cc6.svg
Furong Huang Lab at UMD
user
furongh-lab
65a09d6ce969415381f1a332
Hao Liu
admin_assigned
2024-01-12T10:52:12.162Z
62aa53fc5df2251f7df798c6
/avatars/1c7828ce861c30ddd0afceff355a92ec.svg
Hao
user
haoliu
65a09d6ce969415381f1a333
Heng Ji
65a09d6ce969415381f1a334
Hongyi Wang
admin_assigned
2024-01-12T10:52:53.969Z
64e5414fe56d371e42da69c4
/avatars/7d4a101b0338410bc736e04d7a611aca.svg
Hongyi Wang
user
hwang595
65a09d6ce969415381f1a335
Huan Zhang
65a09d6ce969415381f1a336
Huaxiu Yao
65a09d6ce969415381f1a337
Manolis Kellis
65a09d6ce969415381f1a338
Marinka Zitnik
admin_assigned
2024-01-12T10:54:07.144Z
636826f95bb06007ea0e911e
https://cdn-avatars.huggingface.co/v1/production/uploads/1667770136112-636826f95bb06007ea0e911e.jpeg
Marinka Zitnik
user
marinkaz
65a09d6ce969415381f1a339
Meng Jiang
admin_assigned
2024-01-12T10:54:57.296Z
62cdc931c589e4a9e2411d39
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
Meng Jiang
user
mjiang89
65a09d6ce969415381f1a33a
Mohit Bansal
65a09d6ce969415381f1a33b
James Zou
admin_assigned
2024-01-12T10:55:28.539Z
648a769003fc4a3938bb7943
/avatars/7647f99abdcca4251fcac7783b6fcc8d.svg
zou
user
jameszou707
65a09d6ce969415381f1a33c
Jian Pei
admin_assigned
2024-01-12T10:55:37.180Z
63e9d45fccae1fe5c624704a
/avatars/24644537b352a4e43cc9fd05f954a43c.svg
jianpei.wjp
user
jianpei
65a09d6ce969415381f1a33d
Jian Liu
65a09d6ce969415381f1a33e
Jianfeng Gao
admin_assigned
2024-01-12T10:55:49.797Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
65a09d6ce969415381f1a33f
Jiawei Han
65a09d6ce969415381f1a340
Jieyu Zhao
admin_assigned
2024-01-12T10:56:16.185Z
64ba0230c9a9894feb0cd32b
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba0230c9a9894feb0cd32b/V1XWySRXAhLN4wqXZJ3Ai.jpeg
Jieyu Zhao
user
jieyuz
65a09d6ce969415381f1a341
Jiliang Tang
65a09d6ce969415381f1a342
Jindong Wang
admin_assigned
2024-01-12T10:56:36.627Z
6204cc0d522e40b4a18d86e2
/avatars/18daf2de5671e711dc745388dd60569d.svg
Jindong Wang
user
jindongwang
65a09d6ce969415381f1a343
John Mitchell
65a09d6ce969415381f1a344
Kai Shu
65a09d6ce969415381f1a345
Kaidi Xu
admin_assigned
2024-01-12T10:57:22.933Z
65539ae68771298696b06aaf
/avatars/5240fa09876b90df763aee0e5a1c24e8.svg
Kaidi Xu
user
kaidiXu
65a09d6ce969415381f1a346
Kai-Wei Chang
admin_assigned
2024-01-12T10:57:31.794Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
65a09d6ce969415381f1a347
Lifang He
65a09d6ce969415381f1a348
Lifu Huang
65a09d6ce969415381f1a349
Michael Backes
admin_assigned
2024-01-12T10:58:01.469Z
64d0133a3c4a1b39a0f46288
/avatars/6f187dd88e88caf1fe31127c5d0827c1.svg
Heinrich
user
backm007
65a09d6ce969415381f1a34a
Neil Zhenqiang Gong
65a09d6ce969415381f1a34b
Philip S. Yu
65a09d6ce969415381f1a34c
Pin-Yu Chen
admin_assigned
2024-01-12T10:58:27.459Z
6495dd0b71f6708e0f990032
https://cdn-avatars.huggingface.co/v1/production/uploads/6495dd0b71f6708e0f990032/PBIjdKNnpkxvR_3djCGVm.png
Pin-Yu Chen
user
pinyuchen
65a09d6ce969415381f1a34d
Quanquan Gu
admin_assigned
2024-01-12T10:58:35.118Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
65a09d6ce969415381f1a34e
Ran Xu
claimed_verified
2024-08-19T22:15:22.742Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
65a09d6ce969415381f1a34f
Rex Ying
65a09d6ce969415381f1a350
Shuiwang Ji
65a09d6ce969415381f1a351
Suman Jana
admin_assigned
2024-01-12T10:59:11.212Z
645092f95fdcff143ad9ae84
/avatars/e393586c34ad7e35091db8e13a8f2166.svg
Suman jana
user
Suman116
65a09d6ce969415381f1a352
Tianlong Chen
65a09d6ce969415381f1a353
Tianming Liu
65a09d6ce969415381f1a354
Tianyi Zhou
admin_assigned
2024-01-12T10:59:54.064Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
65a09d6ce969415381f1a355
Willian Wang
65a09d6ce969415381f1a356
Xiang Li
65a09d6ce969415381f1a357
Xiangliang Zhang
admin_assigned
2024-01-12T11:00:18.435Z
605a97d9b54d35bc67a4ff12
/avatars/7a48a2dac4e6ebb9e775022e15ddc5a7.svg
zhangxiangliang
user
ZhangXiangliang
65a09d6ce969415381f1a358
Xiao Wang
65a09d6ce969415381f1a359
Xing Xie
65a09d6ce969415381f1a35a
Xun Chen
65a09d6ce969415381f1a35b
Xuyu Wang
65a09d6ce969415381f1a35c
Yan Liu
65a09d6ce969415381f1a35d
Yanfang Ye
65a09d6ce969415381f1a35e
Yinzhi Cao
65a09d6ce969415381f1a35f
Yue Zhao
TrustLLM: Trustworthiness in Large Language Models
Large language models (LLMs), exemplified by ChatGPT, have gained
considerable attention for their excellent natural language processing
capabilities. Nonetheless, these LLMs present many challenges, particularly in
the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs
emerges as an important topic. This paper introduces TrustLLM, a comprehensive
study of trustworthiness in LLMs, including principles for different dimensions
of trustworthiness, established benchmark, evaluation, and analysis of
trustworthiness for mainstream LLMs, and discussion of open challenges and
future directions. Specifically, we first propose a set of principles for
trustworthy LLMs that span eight different dimensions. Based on these
principles, we further establish a benchmark across six dimensions including
truthfulness, safety, fairness, robustness, privacy, and machine ethics. We
then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of
over 30 datasets. Our findings firstly show that in general trustworthiness and
utility (i.e., functional effectiveness) are positively related. Secondly, our
observations reveal that proprietary LLMs generally outperform most open-source
counterparts in terms of trustworthiness, raising concerns about the potential
risks of widely accessible open-source LLMs. However, a few open-source LLMs
come very close to proprietary ones. Thirdly, it is important to note that some
LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent
that they compromise their utility by mistakenly treating benign prompts as
harmful and consequently not responding. Finally, we emphasize the importance
of ensuring transparency not only in the models themselves but also in the
technologies that underpin trustworthiness. Knowing the specific trustworthy
technologies that have been employed is crucial for analyzing their
effectiveness.
This study assesses the trustworthiness of large language models across various dimensions, including truthfulness, safety, fairness, robustness, privacy, and machine ethics, finding a positive correlation with utility and highlighting differences between proprietary and open-source models.
TrustLLM large language models LLMs truthfulness safety fairness robustness privacy machine ethics evaluation benchmark