title: Benchmarking Chinese Knowledge Rectification in Large Language Models
summary: While Large Language Models (LLMs) exhibit remarkable generative
  capabilities, they are not without flaws, particularly in the form of
  hallucinations. This issue is even more pronounced when LLMs are applied to
  specific languages and domains. For example, LLMs may generate nonsense
  information when handling Chinese ancient poetry, proverbs, or idioms, owing to
  the lack of specific knowledge. To this end, this paper introduces a benchmark
  for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically,
  we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of
  knowledge from various sources, including classical texts, idioms, and content
  from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony,
  antithesis, and logical constructs inherent in the Chinese language. Through
  the analysis of this dataset, we uncover the challenges faced by current LLMs
  in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge
  editing techniques on this dataset unveil the substantial scope for advancement
  in the rectification of Chinese knowledge. Code and dataset are available at
  https://github.com/zjunlp/EasyEdit.
paper_authors._id: 66dfdbee0b0271487130022f
paper_authors.name: Tianhe Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:11:28.765Z
paper_authors._id: 64fd394bc311613ad00c5312
paper_authors.avatarUrl: /avatars/b68c898520dd728874c9c718f68e5eee.svg
paper_authors.fullname: Tianhe Lu
paper_authors.type: user
paper_authors.user: TianheLu
paper_authors._id: 66dfdbee0b02714871300230
paper_authors.name: Jizhan Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:11:35.085Z
paper_authors._id: 669663472d25bd04e9af1d66
paper_authors.avatarUrl: /avatars/8b11d5d79d1b8b205baa498a942f573c.svg
paper_authors.fullname: Jizhan Fang
paper_authors.type: user
paper_authors.user: JizhanFang
paper_authors._id: 66dfdbee0b02714871300231
paper_authors.name: Yunzhi Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:11:41.447Z
paper_authors._id: 6122fbe636a4c36a99dbea7b
paper_authors.avatarUrl: /avatars/c0cd2c1ef58e315d9adda9d26000f625.svg
paper_authors.fullname: Yunzhi Yao
paper_authors.type: user
paper_authors.user: cowTodd
paper_authors._id: 66dfdbee0b02714871300232
paper_authors.name: Xin Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:50:04.566Z
paper_authors._id: 6190ab805ca89a28e9f66873
paper_authors.avatarUrl: /avatars/a677a8401360be473895494e5fb267bb.svg
paper_authors.fullname: Xin Xu
paper_authors.type: user
paper_authors.user: XinXuNLPer
paper_authors._id: 66dfdbee0b02714871300233
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T07:10:26.167Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_authors._id: 66dfdbee0b02714871300234
paper_authors.name: Huajun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:12:06.034Z
paper_authors._id: 64931296137833d7ec7689cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
paper_authors.fullname: Huajun Chen
paper_authors.type: user
paper_authors.user: huajunsir
paper_title: Benchmarking Chinese Knowledge Rectification in Large Language Models
paper_summary: While Large Language Models (LLMs) exhibit remarkable generative
  capabilities, they are not without flaws, particularly in the form of
  hallucinations. This issue is even more pronounced when LLMs are applied to
  specific languages and domains. For example, LLMs may generate nonsense
  information when handling Chinese ancient poetry, proverbs, or idioms, owing to
  the lack of specific knowledge. To this end, this paper introduces a benchmark
  for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically,
  we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of
  knowledge from various sources, including classical texts, idioms, and content
  from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony,
  antithesis, and logical constructs inherent in the Chinese language. Through
  the analysis of this dataset, we uncover the challenges faced by current LLMs
  in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge
  editing techniques on this dataset unveil the substantial scope for advancement
  in the rectification of Chinese knowledge. Code and dataset are available at
  https://github.com/zjunlp/EasyEdit.
paper_ai_summary: A benchmark dataset (CKnowEdit) is introduced to address knowledge gaps in LLMs when generating Chinese content, highlighting areas requiring improvement in knowledge editing techniques.
paper_ai_keywords: Large Language Models (LLMs) hallucinations Chinese knowledge CKnowEdit knowledge editing classical texts idioms Baidu Tieba Ruozhiba polyphony antithesis logical constructs