title: COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for
    Alignment with Human Values
summary: Aligning large language models (LLMs) with human preferences has achieved
  remarkable success. However, existing Chinese preference datasets are limited
  by small scale, narrow domain coverage, and lack of rigorous data validation.
  Additionally, the reliance on human annotators for instruction and response
  labeling significantly constrains the scalability of human preference datasets.
  To address these challenges, we design an LLM-based Chinese preference dataset
  annotation pipeline with no human intervention. Specifically, we crawled and
  carefully filtered 92k high-quality Chinese queries and employed 15 mainstream
  LLMs to generate and score chosen-rejected response pairs. Based on it, we
  introduce COIG-P (Chinese Open Instruction Generalist - Preference), a
  high-quality, large-scale Chinese preference dataset, comprises 1,009k Chinese
  preference pairs spanning 6 diverse domains: Chat, Code, Math, Logic, Novel,
  and Role. Building upon COIG-P, to reduce the overhead of using LLMs for
  scoring, we trained a 8B-sized Chinese Reward Model (CRM) and meticulously
  constructed a Chinese Reward Benchmark (CRBench). Evaluation results based on
  AlignBench liu2024alignbenchbenchmarkingchinesealignment show that that
  COIG-P significantly outperforms other Chinese preference datasets, and it
  brings significant performance improvements ranging from 2% to 12% for the
  Qwen2/2.5 and Infinity-Instruct-3M-0625 model series, respectively. The results
  on CRBench demonstrate that our CRM has a strong and robust scoring ability. We
  apply it to filter chosen-rejected response pairs in a test split of COIG-P,
  and our experiments show that it is comparable to GPT-4o in identifying
  low-quality samples while maintaining efficiency and cost-effectiveness. Our
  codes and data are released in
  https://github.com/multimodal-art-projection/COIG-P.
paper_authors._id: 67f630091aed1b4344b57c1b
paper_authors.name: M-A-P Team
paper_authors._id: 67f630091aed1b4344b57c1c
paper_authors.name: Siwei Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T14:36:41.051Z
paper_authors._id: 656d97b10bbc114fe64a96c5
paper_authors.avatarUrl: /avatars/fd23bae1d85c5b96c42064a5ddcfad41.svg
paper_authors.fullname: SiweiWu
paper_authors.type: user
paper_authors.user: SiweiWu
paper_authors._id: 67f630091aed1b4344b57c1d
paper_authors.name: Jincheng Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T09:48:17.939Z
paper_authors._id: 6704ee27386892c420db1938
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6704ee27386892c420db1938/lb5mtEwYhn47RawkynYPs.jpeg
paper_authors.fullname: JinCheng Ren
paper_authors.type: user
paper_authors.user: JinChengRen
paper_authors._id: 67f630091aed1b4344b57c1e
paper_authors.name: Xinrun Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T14:36:47.680Z
paper_authors._id: 654907a4a1faff97850c4eff
paper_authors.avatarUrl: /avatars/458c90151614bc7f116943b6e67d6b8a.svg
paper_authors.fullname: du
paper_authors.type: user
paper_authors.user: dododododo
paper_authors._id: 67f630091aed1b4344b57c1f
paper_authors.name: Shuyue Guo
paper_authors._id: 67f630091aed1b4344b57c20
paper_authors.name: Xingwei Qu
paper_authors._id: 67f630091aed1b4344b57c21
paper_authors.name: Yiming Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:35:12.067Z
paper_authors._id: 6555e8d8a0c34cd61a6b9ce3
paper_authors.avatarUrl: /avatars/71dc562cef4bd42f6b762f036357c800.svg
paper_authors.fullname: yimingliang
paper_authors.type: user
paper_authors.user: yimingliang
paper_authors._id: 67f630091aed1b4344b57c22
paper_authors.name: Jie Liu
paper_authors._id: 67f630091aed1b4344b57c23
paper_authors.name: Yunwen Li
paper_authors._id: 67f630091aed1b4344b57c24
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T09:48:16.069Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 67f630091aed1b4344b57c25
paper_authors.name: Boyu Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T14:36:53.597Z
paper_authors._id: 67f654ecb88bc093ada9da3f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Anh0oLDzzHCeSmEzwAmYd.png
paper_authors.fullname: boyuFeng
paper_authors.type: user
paper_authors.user: FWORKS
paper_authors._id: 67f630091aed1b4344b57c26
paper_authors.name: Huaqing Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:45:34.650Z
paper_authors._id: 67f6adf26d55263a1b0d1ea9
paper_authors.avatarUrl: /avatars/f8cf3dca6d809518b49f20883b4e24fb.svg
paper_authors.fullname: Huaqing Yuan
paper_authors.type: user
paper_authors.user: huaqingyuan
paper_authors._id: 67f630091aed1b4344b57c27
paper_authors.name: Zenith Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:35:42.141Z
paper_authors._id: 632c59bbea6e62428ab3b480
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632c59bbea6e62428ab3b480/XroXmPnGl1fDrSnN1bQ99.jpeg
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: Zenithwang
paper_authors._id: 67f630091aed1b4344b57c28
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:35:49.044Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 67f630091aed1b4344b57c29
paper_authors.name: Wenhao Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:36:15.438Z
paper_authors._id: 641e5bf65f274a0a92c2f6a2
paper_authors.avatarUrl: /avatars/c15a54c51998c0e6367685e8e1737ec9.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: EZ-hwh
paper_authors._id: 67f630091aed1b4344b57c2a
paper_authors.name: Chenglin Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:36:31.116Z
paper_authors._id: 64f9c21b681224dbe49a2280
paper_authors.avatarUrl: /avatars/df26cc4b4c6105af2c77392db61e3a27.svg
paper_authors.fullname: caichenglin
paper_authors.type: user
paper_authors.user: easy4mego
paper_authors._id: 67f630091aed1b4344b57c2b
paper_authors.name: Haoran Que
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:36:48.325Z
paper_authors._id: 676935578263df95771cb486
paper_authors.avatarUrl: /avatars/f81098cf991ee8bc2cab653fbddd0913.svg
paper_authors.fullname: Haoran Que
paper_authors.type: user
paper_authors.user: Quehry2
paper_authors._id: 67f630091aed1b4344b57c2c
paper_authors.name: Jian Yang
paper_authors._id: 67f630091aed1b4344b57c2d
paper_authors.name: Yuelin Bai
paper_authors._id: 67f630091aed1b4344b57c2e
paper_authors.name: Zekun Moore Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:37:03.321Z
paper_authors._id: 6149a9e95347647e6bb68882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
paper_authors.fullname: Zekun Moore Wang
paper_authors.type: user
paper_authors.user: ZenMoore
paper_authors._id: 67f630091aed1b4344b57c2f
paper_authors.name: Zhouliang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T14:36:43.986Z
paper_authors._id: 62a80fe3ac97233f1625235a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
paper_authors.fullname: Zhouliang Yu
paper_authors.type: user
paper_authors.user: zhouliang
paper_authors._id: 67f630091aed1b4344b57c30
paper_authors.name: Qunshu Lin
paper_authors._id: 67f630091aed1b4344b57c31
paper_authors.name: Ding Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:37:19.142Z
paper_authors._id: 65d5c5d6c79c53aff7420794
paper_authors.avatarUrl: /avatars/f248c4ae2eb022e732b754cfa7235e0a.svg
paper_authors.fullname: DingPan
paper_authors.type: user
paper_authors.user: DingPan
paper_authors._id: 67f630091aed1b4344b57c32
paper_authors.name: Yuchen Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:37:26.925Z
paper_authors._id: 64fff4debef9b5946564d490
paper_authors.avatarUrl: /avatars/c7856111a030b6b8d96650741471f0ce.svg
paper_authors.fullname: Jiang Yuchen
paper_authors.type: user
paper_authors.user: YuChenJiang
paper_authors._id: 67f630091aed1b4344b57c33
paper_authors.name: Tiannan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:37:34.116Z
paper_authors._id: 632bfaebea6e62428ab0e9c2
paper_authors.avatarUrl: /avatars/344aaf371bbba9aea091b12741c451e5.svg
paper_authors.fullname: Tiannan Wang
paper_authors.type: user
paper_authors.user: WTNswaggy
paper_authors._id: 67f630091aed1b4344b57c34
paper_authors.name: Wangchunshu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:37:43.878Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 67f630091aed1b4344b57c35
paper_authors.name: Shenzhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:37:53.643Z
paper_authors._id: 6486dde1f74857df3f1a5828
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
paper_authors.fullname: Shenzhi Wang
paper_authors.type: user
paper_authors.user: shenzhi-wang
paper_authors._id: 67f630091aed1b4344b57c36
paper_authors.name: Xingyuan Bu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T14:36:50.729Z
paper_authors._id: 6444e7765691ca69b0d95856
paper_authors.avatarUrl: /avatars/4ae5001e7c7dea89c4deaf2b05436857.svg
paper_authors.fullname: Xingyuan Bu
paper_authors.type: user
paper_authors.user: sefira32
paper_authors._id: 67f630091aed1b4344b57c37
paper_authors.name: Minghao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T09:48:19.721Z
paper_authors._id: 6417d9ea8f689506e7148417
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
paper_authors.fullname: minghao
paper_authors.type: user
paper_authors.user: Liam-Liu
paper_authors._id: 67f630091aed1b4344b57c38
paper_authors.name: Guoyin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:38:01.661Z
paper_authors._id: 6490d4ba1afdee3acd1147f6
paper_authors.avatarUrl: /avatars/ae13c7b21fe9ced7541dcd664d1b94ed.svg
paper_authors.fullname: Guoyin Wang
paper_authors.type: user
paper_authors.user: guoyinwang
paper_authors._id: 67f630091aed1b4344b57c39
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:38:09.800Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 67f630091aed1b4344b57c3a
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:38:19.130Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_title: COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for
    Alignment with Human Values
paper_summary: Aligning large language models (LLMs) with human preferences has achieved
  remarkable success. However, existing Chinese preference datasets are limited
  by small scale, narrow domain coverage, and lack of rigorous data validation.
  Additionally, the reliance on human annotators for instruction and response
  labeling significantly constrains the scalability of human preference datasets.
  To address these challenges, we design an LLM-based Chinese preference dataset
  annotation pipeline with no human intervention. Specifically, we crawled and
  carefully filtered 92k high-quality Chinese queries and employed 15 mainstream
  LLMs to generate and score chosen-rejected response pairs. Based on it, we
  introduce COIG-P (Chinese Open Instruction Generalist - Preference), a
  high-quality, large-scale Chinese preference dataset, comprises 1,009k Chinese
  preference pairs spanning 6 diverse domains: Chat, Code, Math, Logic, Novel,
  and Role. Building upon COIG-P, to reduce the overhead of using LLMs for
  scoring, we trained a 8B-sized Chinese Reward Model (CRM) and meticulously
  constructed a Chinese Reward Benchmark (CRBench). Evaluation results based on
  AlignBench liu2024alignbenchbenchmarkingchinesealignment show that that
  COIG-P significantly outperforms other Chinese preference datasets, and it
  brings significant performance improvements ranging from 2% to 12% for the
  Qwen2/2.5 and Infinity-Instruct-3M-0625 model series, respectively. The results
  on CRBench demonstrate that our CRM has a strong and robust scoring ability. We
  apply it to filter chosen-rejected response pairs in a test split of COIG-P,
  and our experiments show that it is comparable to GPT-4o in identifying
  low-quality samples while maintaining efficiency and cost-effectiveness. Our
  codes and data are released in
  https://github.com/multimodal-art-projection/COIG-P.
paper_ai_summary: A new Chinese preference dataset and reward model are introduced to scale human preference alignment using LLMs, enhancing performance and cost-effectiveness.
paper_ai_keywords: LLMs preference dataset annotation pipeline COIG-P domains Reward Model CRM Reward Benchmark CRBench AlignBench Qwen2 Infinity-Instruct GPT-4