title: TrustLLM: Trustworthiness in Large Language Models
summary: Large language models (LLMs), exemplified by ChatGPT, have gained
  considerable attention for their excellent natural language processing
  capabilities. Nonetheless, these LLMs present many challenges, particularly in
  the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs
  emerges as an important topic. This paper introduces TrustLLM, a comprehensive
  study of trustworthiness in LLMs, including principles for different dimensions
  of trustworthiness, established benchmark, evaluation, and analysis of
  trustworthiness for mainstream LLMs, and discussion of open challenges and
  future directions. Specifically, we first propose a set of principles for
  trustworthy LLMs that span eight different dimensions. Based on these
  principles, we further establish a benchmark across six dimensions including
  truthfulness, safety, fairness, robustness, privacy, and machine ethics. We
  then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of
  over 30 datasets. Our findings firstly show that in general trustworthiness and
  utility (i.e., functional effectiveness) are positively related. Secondly, our
  observations reveal that proprietary LLMs generally outperform most open-source
  counterparts in terms of trustworthiness, raising concerns about the potential
  risks of widely accessible open-source LLMs. However, a few open-source LLMs
  come very close to proprietary ones. Thirdly, it is important to note that some
  LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent
  that they compromise their utility by mistakenly treating benign prompts as
  harmful and consequently not responding. Finally, we emphasize the importance
  of ensuring transparency not only in the models themselves but also in the
  technologies that underpin trustworthiness. Knowing the specific trustworthy
  technologies that have been employed is crucial for analyzing their
  effectiveness.
paper_authors._id: 65a09d6ce969415381f1a31d
paper_authors.name: Lichao Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:29:16.487Z
paper_authors._id: 65a52766215aabac489e3468
paper_authors.avatarUrl: /avatars/fe05e22cd7e12e961296426434e17c76.svg
paper_authors.fullname: Lichao Sun
paper_authors.type: user
paper_authors.user: sunlichao137
paper_authors._id: 65a09d6ce969415381f1a31e
paper_authors.name: Yue Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:08:53.072Z
paper_authors._id: 639d94ab7145123e0d44e48a
paper_authors.avatarUrl: /avatars/5bb6a65b306d1383c4a8bcd9334b470a.svg
paper_authors.fullname: Yue Huang
paper_authors.type: user
paper_authors.user: HowieHwong
paper_authors._id: 65a09d6ce969415381f1a31f
paper_authors.name: Haoran Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-15T07:42:05.891Z
paper_authors._id: 64b82c659ebb69a79f0073f6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b82c659ebb69a79f0073f6/INEHG7kijEHOhFZMjdBIM.png
paper_authors.fullname: Haoran Wang
paper_authors.type: user
paper_authors.user: wang2226
paper_authors._id: 65a09d6ce969415381f1a320
paper_authors.name: Siyuan Wu
paper_authors._id: 65a09d6ce969415381f1a321
paper_authors.name: Qihui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-12T13:42:12.968Z
paper_authors._id: 65a137d15dce70a302586ec9
paper_authors.avatarUrl: /avatars/4290bc4a6d1a1440b76d9e34ceeea4e3.svg
paper_authors.fullname: Qihui Zhang
paper_authors.type: user
paper_authors.user: 7Hui
paper_authors._id: 65a09d6ce969415381f1a322
paper_authors.name: Chujie Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T10:01:10.373Z
paper_authors._id: 65a13cb1c5770b27aef2a2bc
paper_authors.avatarUrl: /avatars/88ec5b988f10ad9fd4d469ae2fa34680.svg
paper_authors.fullname: Chujie Gao
paper_authors.type: user
paper_authors.user: Flossie
paper_authors._id: 65a09d6ce969415381f1a323
paper_authors.name: Yixin Huang
paper_authors._id: 65a09d6ce969415381f1a324
paper_authors.name: Wenhan Lyu
paper_authors._id: 65a09d6ce969415381f1a325
paper_authors.name: Yixuan Zhang
paper_authors._id: 65a09d6ce969415381f1a326
paper_authors.name: Xiner Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:49:50.423Z
paper_authors._id: 651dc3862c5d36450919d0dc
paper_authors.avatarUrl: /avatars/03fb549171d3b103914aa64103e5739e.svg
paper_authors.fullname: Xiner Li
paper_authors.type: user
paper_authors.user: hyanan16
paper_authors._id: 65a09d6ce969415381f1a327
paper_authors.name: Zhengliang Liu
paper_authors._id: 65a09d6ce969415381f1a328
paper_authors.name: Yixin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:50:25.731Z
paper_authors._id: 6244de1c1c560fb11edfca44
paper_authors.avatarUrl: /avatars/36558928bd04be7f49837d4c603681d7.svg
paper_authors.fullname: Yixin Liu
paper_authors.type: user
paper_authors.user: henryL7
paper_authors._id: 65a09d6ce969415381f1a329
paper_authors.name: Yijue Wang
paper_authors._id: 65a09d6ce969415381f1a32a
paper_authors.name: Zhikun Zhang
paper_authors._id: 65a09d6ce969415381f1a32b
paper_authors.name: Bhavya Kailkhura
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T12:44:31.015Z
paper_authors._id: 65cb79db6427380bc21261e2
paper_authors.avatarUrl: /avatars/a003eb5d0955417329c1a4170ae65879.svg
paper_authors.fullname: Bhavya Kailkhura
paper_authors.type: user
paper_authors.user: bhavyakailkhura
paper_authors._id: 65a09d6ce969415381f1a32c
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:51:13.398Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 65a09d6ce969415381f1a32d
paper_authors.name: Chao Zhang
paper_authors._id: 65a09d6ce969415381f1a32e
paper_authors.name: Chaowei Xiao
paper_authors._id: 65a09d6ce969415381f1a32f
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:51:35.436Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 65a09d6ce969415381f1a330
paper_authors.name: Eric Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:51:42.959Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_authors._id: 65a09d6ce969415381f1a331
paper_authors.name: Furong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:51:53.877Z
paper_authors._id: 64cbc3e2a257a3212c00a115
paper_authors.avatarUrl: /avatars/836e61be4aeda2080ddf2db9f2626cc6.svg
paper_authors.fullname: Furong Huang Lab at UMD
paper_authors.type: user
paper_authors.user: furongh-lab
paper_authors._id: 65a09d6ce969415381f1a332
paper_authors.name: Hao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:52:12.162Z
paper_authors._id: 62aa53fc5df2251f7df798c6
paper_authors.avatarUrl: /avatars/1c7828ce861c30ddd0afceff355a92ec.svg
paper_authors.fullname: Hao
paper_authors.type: user
paper_authors.user: haoliu
paper_authors._id: 65a09d6ce969415381f1a333
paper_authors.name: Heng Ji
paper_authors._id: 65a09d6ce969415381f1a334
paper_authors.name: Hongyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:52:53.969Z
paper_authors._id: 64e5414fe56d371e42da69c4
paper_authors.avatarUrl: /avatars/7d4a101b0338410bc736e04d7a611aca.svg
paper_authors.fullname: Hongyi Wang
paper_authors.type: user
paper_authors.user: hwang595
paper_authors._id: 65a09d6ce969415381f1a335
paper_authors.name: Huan Zhang
paper_authors._id: 65a09d6ce969415381f1a336
paper_authors.name: Huaxiu Yao
paper_authors._id: 65a09d6ce969415381f1a337
paper_authors.name: Manolis Kellis
paper_authors._id: 65a09d6ce969415381f1a338
paper_authors.name: Marinka Zitnik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:54:07.144Z
paper_authors._id: 636826f95bb06007ea0e911e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667770136112-636826f95bb06007ea0e911e.jpeg
paper_authors.fullname: Marinka Zitnik
paper_authors.type: user
paper_authors.user: marinkaz
paper_authors._id: 65a09d6ce969415381f1a339
paper_authors.name: Meng Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:54:57.296Z
paper_authors._id: 62cdc931c589e4a9e2411d39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
paper_authors.fullname: Meng Jiang
paper_authors.type: user
paper_authors.user: mjiang89
paper_authors._id: 65a09d6ce969415381f1a33a
paper_authors.name: Mohit Bansal
paper_authors._id: 65a09d6ce969415381f1a33b
paper_authors.name: James Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:55:28.539Z
paper_authors._id: 648a769003fc4a3938bb7943
paper_authors.avatarUrl: /avatars/7647f99abdcca4251fcac7783b6fcc8d.svg
paper_authors.fullname: zou
paper_authors.type: user
paper_authors.user: jameszou707
paper_authors._id: 65a09d6ce969415381f1a33c
paper_authors.name: Jian Pei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:55:37.180Z
paper_authors._id: 63e9d45fccae1fe5c624704a
paper_authors.avatarUrl: /avatars/24644537b352a4e43cc9fd05f954a43c.svg
paper_authors.fullname: jianpei.wjp
paper_authors.type: user
paper_authors.user: jianpei
paper_authors._id: 65a09d6ce969415381f1a33d
paper_authors.name: Jian Liu
paper_authors._id: 65a09d6ce969415381f1a33e
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:55:49.797Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 65a09d6ce969415381f1a33f
paper_authors.name: Jiawei Han
paper_authors._id: 65a09d6ce969415381f1a340
paper_authors.name: Jieyu Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:56:16.185Z
paper_authors._id: 64ba0230c9a9894feb0cd32b
paper_authors.avatarUrl: /avatars/1a9e6eb47c14fe278b8e1d907d518f46.svg
paper_authors.fullname: Jieyu Zhao
paper_authors.type: user
paper_authors.user: jieyuz
paper_authors._id: 65a09d6ce969415381f1a341
paper_authors.name: Jiliang Tang
paper_authors._id: 65a09d6ce969415381f1a342
paper_authors.name: Jindong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:56:36.627Z
paper_authors._id: 6204cc0d522e40b4a18d86e2
paper_authors.avatarUrl: /avatars/18daf2de5671e711dc745388dd60569d.svg
paper_authors.fullname: Jindong Wang
paper_authors.type: user
paper_authors.user: jindongwang
paper_authors._id: 65a09d6ce969415381f1a343
paper_authors.name: John Mitchell
paper_authors._id: 65a09d6ce969415381f1a344
paper_authors.name: Kai Shu
paper_authors._id: 65a09d6ce969415381f1a345
paper_authors.name: Kaidi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:57:22.933Z
paper_authors._id: 65539ae68771298696b06aaf
paper_authors.avatarUrl: /avatars/5240fa09876b90df763aee0e5a1c24e8.svg
paper_authors.fullname: Kaidi Xu
paper_authors.type: user
paper_authors.user: kaidiXu
paper_authors._id: 65a09d6ce969415381f1a346
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:57:31.794Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 65a09d6ce969415381f1a347
paper_authors.name: Lifang He
paper_authors._id: 65a09d6ce969415381f1a348
paper_authors.name: Lifu Huang
paper_authors._id: 65a09d6ce969415381f1a349
paper_authors.name: Michael Backes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:58:01.469Z
paper_authors._id: 64d0133a3c4a1b39a0f46288
paper_authors.avatarUrl: /avatars/6f187dd88e88caf1fe31127c5d0827c1.svg
paper_authors.fullname: Heinrich
paper_authors.type: user
paper_authors.user: backm007
paper_authors._id: 65a09d6ce969415381f1a34a
paper_authors.name: Neil Zhenqiang Gong
paper_authors._id: 65a09d6ce969415381f1a34b
paper_authors.name: Philip S. Yu
paper_authors._id: 65a09d6ce969415381f1a34c
paper_authors.name: Pin-Yu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:58:27.459Z
paper_authors._id: 6495dd0b71f6708e0f990032
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6495dd0b71f6708e0f990032/PBIjdKNnpkxvR_3djCGVm.png
paper_authors.fullname: Pin-Yu Chen
paper_authors.type: user
paper_authors.user: pinyuchen
paper_authors._id: 65a09d6ce969415381f1a34d
paper_authors.name: Quanquan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:58:35.118Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_authors._id: 65a09d6ce969415381f1a34e
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:15:22.742Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_authors._id: 65a09d6ce969415381f1a34f
paper_authors.name: Rex Ying
paper_authors._id: 65a09d6ce969415381f1a350
paper_authors.name: Shuiwang Ji
paper_authors._id: 65a09d6ce969415381f1a351
paper_authors.name: Suman Jana
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:59:11.212Z
paper_authors._id: 645092f95fdcff143ad9ae84
paper_authors.avatarUrl: /avatars/e393586c34ad7e35091db8e13a8f2166.svg
paper_authors.fullname: Suman jana
paper_authors.type: user
paper_authors.user: Suman116
paper_authors._id: 65a09d6ce969415381f1a352
paper_authors.name: Tianlong Chen
paper_authors._id: 65a09d6ce969415381f1a353
paper_authors.name: Tianming Liu
paper_authors._id: 65a09d6ce969415381f1a354
paper_authors.name: Tianyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:59:54.064Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 65a09d6ce969415381f1a355
paper_authors.name: Willian Wang
paper_authors._id: 65a09d6ce969415381f1a356
paper_authors.name: Xiang Li
paper_authors._id: 65a09d6ce969415381f1a357
paper_authors.name: Xiangliang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T11:00:18.435Z
paper_authors._id: 605a97d9b54d35bc67a4ff12
paper_authors.avatarUrl: /avatars/7a48a2dac4e6ebb9e775022e15ddc5a7.svg
paper_authors.fullname: zhangxiangliang
paper_authors.type: user
paper_authors.user: ZhangXiangliang
paper_authors._id: 65a09d6ce969415381f1a358
paper_authors.name: Xiao Wang
paper_authors._id: 65a09d6ce969415381f1a359
paper_authors.name: Xing Xie
paper_authors._id: 65a09d6ce969415381f1a35a
paper_authors.name: Xun Chen
paper_authors._id: 65a09d6ce969415381f1a35b
paper_authors.name: Xuyu Wang
paper_authors._id: 65a09d6ce969415381f1a35c
paper_authors.name: Yan Liu
paper_authors._id: 65a09d6ce969415381f1a35d
paper_authors.name: Yanfang Ye
paper_authors._id: 65a09d6ce969415381f1a35e
paper_authors.name: Yinzhi Cao
paper_authors._id: 65a09d6ce969415381f1a35f
paper_authors.name: Yue Zhao
paper_title: TrustLLM: Trustworthiness in Large Language Models
paper_summary: Large language models (LLMs), exemplified by ChatGPT, have gained
  considerable attention for their excellent natural language processing
  capabilities. Nonetheless, these LLMs present many challenges, particularly in
  the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs
  emerges as an important topic. This paper introduces TrustLLM, a comprehensive
  study of trustworthiness in LLMs, including principles for different dimensions
  of trustworthiness, established benchmark, evaluation, and analysis of
  trustworthiness for mainstream LLMs, and discussion of open challenges and
  future directions. Specifically, we first propose a set of principles for
  trustworthy LLMs that span eight different dimensions. Based on these
  principles, we further establish a benchmark across six dimensions including
  truthfulness, safety, fairness, robustness, privacy, and machine ethics. We
  then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of
  over 30 datasets. Our findings firstly show that in general trustworthiness and
  utility (i.e., functional effectiveness) are positively related. Secondly, our
  observations reveal that proprietary LLMs generally outperform most open-source
  counterparts in terms of trustworthiness, raising concerns about the potential
  risks of widely accessible open-source LLMs. However, a few open-source LLMs
  come very close to proprietary ones. Thirdly, it is important to note that some
  LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent
  that they compromise their utility by mistakenly treating benign prompts as
  harmful and consequently not responding. Finally, we emphasize the importance
  of ensuring transparency not only in the models themselves but also in the
  technologies that underpin trustworthiness. Knowing the specific trustworthy
  technologies that have been employed is crucial for analyzing their
  effectiveness.