title: CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding
    Capabilities of CodeLLMs
summary: Recent advancements in Code Large Language Models (CodeLLMs) have
  predominantly focused on open-ended code generation tasks, often neglecting the
  critical aspect of code understanding and comprehension. To bridge this gap, we
  present CodeMMLU, a comprehensive multiple-choice question-answer benchmark
  designed to evaluate the depth of software and code understanding in LLMs.
  CodeMMLU includes over 10,000 questions sourced from diverse domains,
  encompassing tasks such as code analysis, defect detection, and software
  engineering principles across multiple programming languages. Unlike
  traditional benchmarks, CodeMMLU assesses models's ability to reason about code
  rather than merely generate it, providing deeper insights into their grasp of
  complex software concepts and systems. Our extensive evaluation reveals that
  even state-of-the-art models face significant challenges with CodeMMLU,
  highlighting deficiencies in comprehension beyond code generation. By
  underscoring the crucial relationship between code understanding and effective
  generation, CodeMMLU serves as a vital resource for advancing AI-assisted
  software development, ultimately aiming to create more reliable and capable
  coding assistants.
paper_authors._id: 66ffb6ae03aaa32983158167
paper_authors.name: Dung Nguyen Manh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:29.107Z
paper_authors._id: 63733f7fd398fce0dd45125c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63733f7fd398fce0dd45125c/VzqcN9YCjMDekopqyDS-y.png
paper_authors.fullname: Nguyen Manh Dung
paper_authors.type: user
paper_authors.user: nmd2k
paper_authors._id: 66ffb6ae03aaa32983158168
paper_authors.name: Thang Phan Chau
paper_authors._id: 66ffb6ae03aaa32983158169
paper_authors.name: Nam Le Hai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T08:17:32.517Z
paper_authors._id: 63a27e21f30c4642278de3ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a27e21f30c4642278de3ea/zj0mGDv9DEK9JYHFjfaZq.jpeg
paper_authors.fullname: Le Hai Nam
paper_authors.type: user
paper_authors.user: NamCyan
paper_authors._id: 66ffb6ae03aaa3298315816a
paper_authors.name: Thong T. Doan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T08:17:28.637Z
paper_authors._id: 63e8ad2accae1fe5c61878d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676193023976-noauth.jpeg
paper_authors.fullname: Doan Tien Thong
paper_authors.type: user
paper_authors.user: doantienthongbku
paper_authors._id: 66ffb6ae03aaa3298315816b
paper_authors.name: Nam V. Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T08:17:30.357Z
paper_authors._id: 64c2bea2ada7df214276913b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c2bea2ada7df214276913b/QFCtmCn439Afsr7uqyoMT.jpeg
paper_authors.fullname: Nguyen Van Nam
paper_authors.type: user
paper_authors.user: DavidNguyen
paper_authors._id: 66ffb6ae03aaa3298315816c
paper_authors.name: Quang Pham
paper_authors._id: 66ffb6ae03aaa3298315816d
paper_authors.name: Nghi D. Q. Bui
paper_title: CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding
    Capabilities of CodeLLMs
paper_summary: Recent advancements in Code Large Language Models (CodeLLMs) have
  predominantly focused on open-ended code generation tasks, often neglecting the
  critical aspect of code understanding and comprehension. To bridge this gap, we
  present CodeMMLU, a comprehensive multiple-choice question-answer benchmark
  designed to evaluate the depth of software and code understanding in LLMs.
  CodeMMLU includes over 10,000 questions sourced from diverse domains,
  encompassing tasks such as code analysis, defect detection, and software
  engineering principles across multiple programming languages. Unlike
  traditional benchmarks, CodeMMLU assesses models's ability to reason about code
  rather than merely generate it, providing deeper insights into their grasp of
  complex software concepts and systems. Our extensive evaluation reveals that
  even state-of-the-art models face significant challenges with CodeMMLU,
  highlighting deficiencies in comprehension beyond code generation. By
  underscoring the crucial relationship between code understanding and effective
  generation, CodeMMLU serves as a vital resource for advancing AI-assisted
  software development, ultimately aiming to create more reliable and capable
  coding assistants.
paper_ai_keywords: Code Large Language Models (CodeLLMs) code understanding code comprehension CodeMMLU multiple-choice question-answer benchmark code analysis defect detection software engineering principles programmig languages reasoning about code software concepts software systems AI-assisted software development coding assistants