LONGCODEU: Benchmarking Long-Context Language Models on Long Code
  Understanding
Current advanced long-context language models offer great potential for
real-world software engineering applications. However, progress in this
critical domain remains hampered by a fundamental limitation: the absence of a
rigorous evaluation framework for long code understanding. To gap this
obstacle, we propose a long code understanding benchmark LONGCODEU from four
aspects (8 tasks) to evaluate LCLMs' long code understanding ability required
for practical applications, including code unit perception, intra-code unit
understanding, inter-code unit relation understanding, and long code
documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6
general models and 3 code models). Our experimental results reveal key
limitations in current LCLMs' capabilities for long code understanding.
Particularly, the performance of LCLMs drops dramatically when the long code
length is greater than 32K, falling far short of their claimed 128K-1M context
windows. In the four aspects, inter-code unit relation understanding is the
most challenging for LCLMs. Our study provides valuable insights for optimizing
LCLMs and driving advancements in software engineering.
67ced7aa502759f20555f68d
Jia Li
67ced7aa502759f20555f68e
Xuyuan Guo
67ced7aa502759f20555f68f
Lei Li
claimed_verified
2025-05-13T08:13:36.447Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
67ced7aa502759f20555f690
Kechi Zhang
67ced7aa502759f20555f691
Ge Li
67ced7aa502759f20555f692
Jia Li
67ced7aa502759f20555f693
Zhengwei Tao
67ced7aa502759f20555f694
Fang Liu
67ced7aa502759f20555f695
Chongyang Tao
67ced7aa502759f20555f696
Yuqi Zhu
67ced7aa502759f20555f697
Zhi Jin
LONGCODEU: Benchmarking Long-Context Language Models on Long Code
  Understanding
Current advanced long-context language models offer great potential for
real-world software engineering applications. However, progress in this
critical domain remains hampered by a fundamental limitation: the absence of a
rigorous evaluation framework for long code understanding. To gap this
obstacle, we propose a long code understanding benchmark LONGCODEU from four
aspects (8 tasks) to evaluate LCLMs' long code understanding ability required
for practical applications, including code unit perception, intra-code unit
understanding, inter-code unit relation understanding, and long code
documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6
general models and 3 code models). Our experimental results reveal key
limitations in current LCLMs' capabilities for long code understanding.
Particularly, the performance of LCLMs drops dramatically when the long code
length is greater than 32K, falling far short of their claimed 128K-1M context
windows. In the four aspects, inter-code unit relation understanding is the
most challenging for LCLMs. Our study provides valuable insights for optimizing
LCLMs and driving advancements in software engineering.
A benchmark for evaluating long code understanding in advanced language models identifies key limitations and highlights the need for optimization.
long-context language models LCLMs code unit perception intra-code unit understanding inter-code unit relation understanding long code documentation understanding