title: Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language
    Models' Alignment
summary: Ensuring alignment, which refers to making models behave in accordance with
  human intentions [1,2], has become a critical task before deploying large
  language models (LLMs) in real-world applications. For instance, OpenAI devoted
  six months to iteratively aligning GPT-4 before its release [3]. However, a
  major challenge faced by practitioners is the lack of clear guidance on
  evaluating whether LLM outputs align with social norms, values, and
  regulations. This obstacle hinders systematic iteration and deployment of LLMs.
  To address this issue, this paper presents a comprehensive survey of key
  dimensions that are crucial to consider when assessing LLM trustworthiness. The
  survey covers seven major categories of LLM trustworthiness: reliability,
  safety, fairness, resistance to misuse, explainability and reasoning, adherence
  to social norms, and robustness. Each major category is further divided into
  several sub-categories, resulting in a total of 29 sub-categories.
  Additionally, a subset of 8 sub-categories is selected for further
  investigation, where corresponding measurement studies are designed and
  conducted on several widely-used LLMs. The measurement results indicate that,
  in general, more aligned models tend to perform better in terms of overall
  trustworthiness. However, the effectiveness of alignment varies across the
  different trustworthiness categories considered. This highlights the importance
  of conducting more fine-grained analyses, testing, and making continuous
  improvements on LLM alignment. By shedding light on these key dimensions of LLM
  trustworthiness, this paper aims to provide valuable insights and guidance to
  practitioners in the field. Understanding and addressing these concerns will be
  crucial in achieving reliable and ethically sound deployment of LLMs in various
  applications.
paper_authors._id: 64d595fe02e58cc1fdd693d4
paper_authors.name: Yang Liu
paper_authors._id: 64d595fe02e58cc1fdd693d5
paper_authors.name: Yuanshun Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-13T00:11:48.877Z
paper_authors._id: 642f5cd0ee432fcf2e91fbb2
paper_authors.avatarUrl: /avatars/f8021d782425bb6bb1c8b70da5f73e55.svg
paper_authors.fullname: Kevin Yao
paper_authors.type: user
paper_authors.user: kevinyaoresearch
paper_authors._id: 64d595fe02e58cc1fdd693d6
paper_authors.name: Jean-Francois Ton
paper_authors._id: 64d595fe02e58cc1fdd693d7
paper_authors.name: Xiaoying Zhang
paper_authors._id: 64d595fe02e58cc1fdd693d8
paper_authors.name: Ruocheng Guo Hao Cheng
paper_authors._id: 64d595fe02e58cc1fdd693d9
paper_authors.name: Yegor Klochkov
paper_authors._id: 64d595fe02e58cc1fdd693da
paper_authors.name: Muhammad Faaiz Taufiq
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-11T14:37:53.443Z
paper_authors._id: 64d6459436b05ef8ffc7cbd1
paper_authors.avatarUrl: /avatars/48c28be2d69ad39e166b82560ee8365a.svg
paper_authors.fullname: Muhammad Faaiz Taufiq
paper_authors.type: user
paper_authors.user: faaizt
paper_authors._id: 64d595fe02e58cc1fdd693db
paper_authors.name: Hang Li
paper_title: Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language
    Models' Alignment
paper_summary: Ensuring alignment, which refers to making models behave in accordance with
  human intentions [1,2], has become a critical task before deploying large
  language models (LLMs) in real-world applications. For instance, OpenAI devoted
  six months to iteratively aligning GPT-4 before its release [3]. However, a
  major challenge faced by practitioners is the lack of clear guidance on
  evaluating whether LLM outputs align with social norms, values, and
  regulations. This obstacle hinders systematic iteration and deployment of LLMs.
  To address this issue, this paper presents a comprehensive survey of key
  dimensions that are crucial to consider when assessing LLM trustworthiness. The
  survey covers seven major categories of LLM trustworthiness: reliability,
  safety, fairness, resistance to misuse, explainability and reasoning, adherence
  to social norms, and robustness. Each major category is further divided into
  several sub-categories, resulting in a total of 29 sub-categories.
  Additionally, a subset of 8 sub-categories is selected for further
  investigation, where corresponding measurement studies are designed and
  conducted on several widely-used LLMs. The measurement results indicate that,
  in general, more aligned models tend to perform better in terms of overall
  trustworthiness. However, the effectiveness of alignment varies across the
  different trustworthiness categories considered. This highlights the importance
  of conducting more fine-grained analyses, testing, and making continuous
  improvements on LLM alignment. By shedding light on these key dimensions of LLM
  trustworthiness, this paper aims to provide valuable insights and guidance to
  practitioners in the field. Understanding and addressing these concerns will be
  crucial in achieving reliable and ethically sound deployment of LLMs in various
  applications.
paper_ai_summary: This survey examines key dimensions of trustworthiness for large language models, including reliability, safety, fairness, misuse resistance, explainability, social norms adherence, and robustness, providing guidelines for evaluating and improving model alignment.
paper_ai_keywords: large language models LLMs alignment reliability safety fairness misuse resistance explainability social norms robustness trustworthiness