title: CS-Bench: A Comprehensive Benchmark for Large Language Models towards
    Computer Science Mastery
summary: Computer Science (CS) stands as a testament to the intricacies of human
  intelligence, profoundly advancing the development of artificial intelligence
  and modern society. However, the current community of large language models
  (LLMs) overly focuses on benchmarks for analyzing specific foundational skills
  (e.g. mathematics and code generation), neglecting an all-round evaluation of
  the computer science field. To bridge this gap, we introduce CS-Bench, the
  first bilingual (Chinese-English) benchmark dedicated to evaluating the
  performance of LLMs in computer science. CS-Bench comprises approximately 5K
  meticulously curated test samples, covering 26 subfields across 4 key areas of
  computer science, encompassing various task forms and divisions of knowledge
  and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of
  over 30 mainstream LLMs, revealing the relationship between CS performance and
  model scales. We also quantitatively analyze the reasons for failures in
  existing LLMs and highlight directions for improvements, including knowledge
  supplementation and CS-specific reasoning. Further cross-capability experiments
  show a high correlation between LLMs' capabilities in computer science and
  their abilities in mathematics and coding. Moreover, expert LLMs specialized in
  mathematics and coding also demonstrate strong performances in several CS
  subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM
  applications in the CS field and paving new avenues in assessing LLMs' diverse
  reasoning capabilities. The CS-Bench data and evaluation code are available at
  https://github.com/csbench/csbench.
paper_authors._id: 666bb1e1fba6c458fcf39d70
paper_authors.name: Xiaoshuai Song
paper_authors._id: 666bb1e1fba6c458fcf39d71
paper_authors.name: Muxi Diao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:40:27.978Z
paper_authors._id: 6768c97367e4b4606a3c9cec
paper_authors.avatarUrl: /avatars/5ddafe7a05828366f66c79072556f370.svg
paper_authors.fullname: diaomuxi
paper_authors.type: user
paper_authors.user: diaomuxi
paper_authors._id: 666bb1e1fba6c458fcf39d72
paper_authors.name: Guanting Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-30T20:58:07.854Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 666bb1e1fba6c458fcf39d73
paper_authors.name: Zhengyang Wang
paper_authors._id: 666bb1e1fba6c458fcf39d74
paper_authors.name: Yujia Fu
paper_authors._id: 666bb1e1fba6c458fcf39d75
paper_authors.name: Runqi Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:52:23.974Z
paper_authors._id: 6683a05e74fb1736a4b7c934
paper_authors.avatarUrl: /avatars/35292253cf1882080be353c4b8c9ca3b.svg
paper_authors.fullname: QRQ
paper_authors.type: user
paper_authors.user: RichardQRQ
paper_authors._id: 666bb1e1fba6c458fcf39d76
paper_authors.name: Zhexu Wang
paper_authors._id: 666bb1e1fba6c458fcf39d77
paper_authors.name: Dayuan Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T12:01:07.517Z
paper_authors._id: 65572fd2bcdc315c0a1395d7
paper_authors.avatarUrl: /avatars/36135ca35bbd4baca37a9092f3e2d82c.svg
paper_authors.fullname: Fu-Dayuan
paper_authors.type: user
paper_authors.user: fudayuan
paper_authors._id: 666bb1e1fba6c458fcf39d78
paper_authors.name: Huangxuan Wu
paper_authors._id: 666bb1e1fba6c458fcf39d79
paper_authors.name: Bin Liang
paper_authors._id: 666bb1e1fba6c458fcf39d7a
paper_authors.name: Weihao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:33.995Z
paper_authors._id: 62751082b43ccfeef483424f
paper_authors.avatarUrl: /avatars/fec83e4478e7d1731ba6033328131852.svg
paper_authors.fullname: WeihaoZeng
paper_authors.type: user
paper_authors.user: AndrewZeng
paper_authors._id: 666bb1e1fba6c458fcf39d7b
paper_authors.name: Yejie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T10:58:14.074Z
paper_authors._id: 6342dc0ee2647466b42918ab
paper_authors.avatarUrl: /avatars/a80a3df2f67410662bf9681ed8834b17.svg
paper_authors.fullname: Yejie Wang
paper_authors.type: user
paper_authors.user: banksy235
paper_authors._id: 666bb1e1fba6c458fcf39d7c
paper_authors.name: Zhuoma GongQue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:34:33.573Z
paper_authors._id: 6762b973686e574cab4717ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/icacS5LkbX2S46b3Dqbni.png
paper_authors.fullname: zhuoma
paper_authors.type: user
paper_authors.user: Ethereal1l
paper_authors._id: 666bb1e1fba6c458fcf39d7d
paper_authors.name: Jianing Yu
paper_authors._id: 666bb1e1fba6c458fcf39d7e
paper_authors.name: Qiuna Tan
paper_authors._id: 666bb1e1fba6c458fcf39d7f
paper_authors.name: Weiran Xu
paper_title: CS-Bench: A Comprehensive Benchmark for Large Language Models towards
    Computer Science Mastery
paper_summary: Computer Science (CS) stands as a testament to the intricacies of human
  intelligence, profoundly advancing the development of artificial intelligence
  and modern society. However, the current community of large language models
  (LLMs) overly focuses on benchmarks for analyzing specific foundational skills
  (e.g. mathematics and code generation), neglecting an all-round evaluation of
  the computer science field. To bridge this gap, we introduce CS-Bench, the
  first bilingual (Chinese-English) benchmark dedicated to evaluating the
  performance of LLMs in computer science. CS-Bench comprises approximately 5K
  meticulously curated test samples, covering 26 subfields across 4 key areas of
  computer science, encompassing various task forms and divisions of knowledge
  and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of
  over 30 mainstream LLMs, revealing the relationship between CS performance and
  model scales. We also quantitatively analyze the reasons for failures in
  existing LLMs and highlight directions for improvements, including knowledge
  supplementation and CS-specific reasoning. Further cross-capability experiments
  show a high correlation between LLMs' capabilities in computer science and
  their abilities in mathematics and coding. Moreover, expert LLMs specialized in
  mathematics and coding also demonstrate strong performances in several CS
  subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM
  applications in the CS field and paving new avenues in assessing LLMs' diverse
  reasoning capabilities. The CS-Bench data and evaluation code are available at
  https://github.com/csbench/csbench.
paper_ai_summary: CS-Bench evaluates large language models across various subfields of computer science, uncovering performance trends and pinpointing areas for improvement.
paper_ai_keywords: large language models LLMs CS-Bench bilingual benchmark computer science test samples subfields knowledge supplementation CS-specific reasoning diverse reasoning capabilities