Benchmarking Chinese Knowledge Rectification in Large Language Models
While Large Language Models (LLMs) exhibit remarkable generative
capabilities, they are not without flaws, particularly in the form of
hallucinations. This issue is even more pronounced when LLMs are applied to
specific languages and domains. For example, LLMs may generate nonsense
information when handling Chinese ancient poetry, proverbs, or idioms, owing to
the lack of specific knowledge. To this end, this paper introduces a benchmark
for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically,
we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of
knowledge from various sources, including classical texts, idioms, and content
from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony,
antithesis, and logical constructs inherent in the Chinese language. Through
the analysis of this dataset, we uncover the challenges faced by current LLMs
in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge
editing techniques on this dataset unveil the substantial scope for advancement
in the rectification of Chinese knowledge. Code and dataset are available at
https://github.com/zjunlp/EasyEdit.
66dfdbee0b0271487130022f
Tianhe Lu
admin_assigned
2024-09-10T08:11:28.765Z
64fd394bc311613ad00c5312
/avatars/b68c898520dd728874c9c718f68e5eee.svg
Tianhe Lu
user
TianheLu
66dfdbee0b02714871300230
Jizhan Fang
admin_assigned
2024-09-10T08:11:35.085Z
669663472d25bd04e9af1d66
/avatars/8b11d5d79d1b8b205baa498a942f573c.svg
Jizhan Fang
user
JizhanFang
66dfdbee0b02714871300231
Yunzhi Yao
admin_assigned
2024-09-10T08:11:41.447Z
6122fbe636a4c36a99dbea7b
/avatars/c0cd2c1ef58e315d9adda9d26000f625.svg
Yunzhi Yao
user
cowTodd
66dfdbee0b02714871300232
Xin Xu
claimed_verified
2024-10-21T07:50:04.566Z
6190ab805ca89a28e9f66873
/avatars/3c7ecc398fbf851acd2a132e947a92be.svg
Xin Xu
user
XinXuNLPer
66dfdbee0b02714871300233
Ningyu Zhang
claimed_verified
2024-09-10T07:10:26.167Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
66dfdbee0b02714871300234
Huajun Chen
admin_assigned
2024-09-10T08:12:06.034Z
64931296137833d7ec7689cd
https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
Huajun Chen
user
huajunsir
Benchmarking Chinese Knowledge Rectification in Large Language Models
While Large Language Models (LLMs) exhibit remarkable generative
capabilities, they are not without flaws, particularly in the form of
hallucinations. This issue is even more pronounced when LLMs are applied to
specific languages and domains. For example, LLMs may generate nonsense
information when handling Chinese ancient poetry, proverbs, or idioms, owing to
the lack of specific knowledge. To this end, this paper introduces a benchmark
for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically,
we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of
knowledge from various sources, including classical texts, idioms, and content
from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony,
antithesis, and logical constructs inherent in the Chinese language. Through
the analysis of this dataset, we uncover the challenges faced by current LLMs
in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge
editing techniques on this dataset unveil the substantial scope for advancement
in the rectification of Chinese knowledge. Code and dataset are available at
https://github.com/zjunlp/EasyEdit.
A benchmark dataset (CKnowEdit) is introduced to address knowledge gaps in LLMs when generating Chinese content, highlighting areas requiring improvement in knowledge editing techniques.
Large Language Models (LLMs) hallucinations Chinese knowledge CKnowEdit knowledge editing classical texts idioms Baidu Tieba Ruozhiba polyphony antithesis logical constructs