title: Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language
    Models
summary: New LLM evaluation benchmarks are important to align with the rapid
  development of Large Language Models (LLMs). In this work, we present Chinese
  SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality
  ability of language models to answer short questions, and Chinese SimpleQA
  mainly has five properties (i.e., Chinese, Diverse, High-quality, Static,
  Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6
  major topics with 99 diverse subtopics. Second, we conduct a comprehensive
  quality control process to achieve high-quality questions and answers, where
  the reference answers are static and cannot be changed over time. Third,
  following SimpleQA, the questions and answers are very short, and the grading
  process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we
  perform a comprehensive evaluation on the factuality abilities of existing
  LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to
  better understand the Chinese factuality abilities of their models and
  facilitate the growth of foundation models.
paper_authors._id: 6732da1ff014ee495569041b
paper_authors.name: Yancheng He
paper_authors._id: 6732da1ff014ee495569041c
paper_authors.name: Shilong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:05:05.132Z
paper_authors._id: 64d8cfbe99eb22b8da673150
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/9gXtURPqQRzR4cyUiDkHT.jpeg
paper_authors.fullname: Shilong Li
paper_authors.type: user
paper_authors.user: w332323
paper_authors._id: 6732da1ff014ee495569041d
paper_authors.name: Jiaheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:03:18.791Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 6732da1ff014ee495569041e
paper_authors.name: Yingshui Tan
paper_authors._id: 6732da1ff014ee495569041f
paper_authors.name: Hui Huang
paper_authors._id: 6732da1ff014ee4955690420
paper_authors.name: Weixun Wang
paper_authors._id: 6732da1ff014ee4955690421
paper_authors.name: Xingyuan Bu
paper_authors._id: 6732da1ff014ee4955690422
paper_authors.name: Hangyu Guo
paper_authors._id: 6732da1ff014ee4955690423
paper_authors.name: Chengwei Hu
paper_authors._id: 6732da1ff014ee4955690424
paper_authors.name: Boren Zheng
paper_authors._id: 6732da1ff014ee4955690425
paper_authors.name: Xuepeng Liu
paper_authors._id: 6732da1ff014ee4955690426
paper_authors.name: Dekai Sun
paper_authors._id: 6732da1ff014ee4955690427
paper_authors.name: Wenbo Su
paper_authors._id: 6732da1ff014ee4955690428
paper_authors.name: Bo Zheng
paper_title: Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language
    Models
paper_summary: New LLM evaluation benchmarks are important to align with the rapid
  development of Large Language Models (LLMs). In this work, we present Chinese
  SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality
  ability of language models to answer short questions, and Chinese SimpleQA
  mainly has five properties (i.e., Chinese, Diverse, High-quality, Static,
  Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6
  major topics with 99 diverse subtopics. Second, we conduct a comprehensive
  quality control process to achieve high-quality questions and answers, where
  the reference answers are static and cannot be changed over time. Third,
  following SimpleQA, the questions and answers are very short, and the grading
  process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we
  perform a comprehensive evaluation on the factuality abilities of existing
  LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to
  better understand the Chinese factuality abilities of their models and
  facilitate the growth of foundation models.