title: LONGCODEU: Benchmarking Long-Context Language Models on Long Code
    Understanding
summary: Current advanced long-context language models offer great potential for
  real-world software engineering applications. However, progress in this
  critical domain remains hampered by a fundamental limitation: the absence of a
  rigorous evaluation framework for long code understanding. To gap this
  obstacle, we propose a long code understanding benchmark LONGCODEU from four
  aspects (8 tasks) to evaluate LCLMs' long code understanding ability required
  for practical applications, including code unit perception, intra-code unit
  understanding, inter-code unit relation understanding, and long code
  documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6
  general models and 3 code models). Our experimental results reveal key
  limitations in current LCLMs' capabilities for long code understanding.
  Particularly, the performance of LCLMs drops dramatically when the long code
  length is greater than 32K, falling far short of their claimed 128K-1M context
  windows. In the four aspects, inter-code unit relation understanding is the
  most challenging for LCLMs. Our study provides valuable insights for optimizing
  LCLMs and driving advancements in software engineering.
paper_authors._id: 67ced7aa502759f20555f68d
paper_authors.name: Jia Li
paper_authors._id: 67ced7aa502759f20555f68e
paper_authors.name: Xuyuan Guo
paper_authors._id: 67ced7aa502759f20555f68f
paper_authors.name: Lei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:13:36.447Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 67ced7aa502759f20555f690
paper_authors.name: Kechi Zhang
paper_authors._id: 67ced7aa502759f20555f691
paper_authors.name: Ge Li
paper_authors._id: 67ced7aa502759f20555f692
paper_authors.name: Jia Li
paper_authors._id: 67ced7aa502759f20555f693
paper_authors.name: Zhengwei Tao
paper_authors._id: 67ced7aa502759f20555f694
paper_authors.name: Fang Liu
paper_authors._id: 67ced7aa502759f20555f695
paper_authors.name: Chongyang Tao
paper_authors._id: 67ced7aa502759f20555f696
paper_authors.name: Yuqi Zhu
paper_authors._id: 67ced7aa502759f20555f697
paper_authors.name: Zhi Jin
paper_title: LONGCODEU: Benchmarking Long-Context Language Models on Long Code
    Understanding
paper_summary: Current advanced long-context language models offer great potential for
  real-world software engineering applications. However, progress in this
  critical domain remains hampered by a fundamental limitation: the absence of a
  rigorous evaluation framework for long code understanding. To gap this
  obstacle, we propose a long code understanding benchmark LONGCODEU from four
  aspects (8 tasks) to evaluate LCLMs' long code understanding ability required
  for practical applications, including code unit perception, intra-code unit
  understanding, inter-code unit relation understanding, and long code
  documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6
  general models and 3 code models). Our experimental results reveal key
  limitations in current LCLMs' capabilities for long code understanding.
  Particularly, the performance of LCLMs drops dramatically when the long code
  length is greater than 32K, falling far short of their claimed 128K-1M context
  windows. In the four aspects, inter-code unit relation understanding is the
  most challenging for LCLMs. Our study provides valuable insights for optimizing
  LCLMs and driving advancements in software engineering.
paper_ai_summary: A benchmark for evaluating long code understanding in advanced language models identifies key limitations and highlights the need for optimization.
paper_ai_keywords: long-context language models LCLMs code unit perception intra-code unit understanding inter-code unit relation understanding long code documentation understanding