Long Code Arena: a Set of Benchmarks for Long-Context Code Models
Nowadays, the fields of code and natural language processing are evolving
rapidly. In particular, models become better at processing long context windows
- supported context sizes have increased by orders of magnitude over the last
few years. However, there is a shortage of benchmarks for code processing that
go beyond a single file of context, while the most popular ones are limited to
a single method. With this work, we aim to close this gap by introducing Long
Code Arena, a suite of six benchmarks for code processing tasks that require
project-wide context. These tasks cover different aspects of code processing:
library-based code generation, CI builds repair, project-level code completion,
commit message generation, bug localization, and module summarization. For each
task, we provide a manually verified dataset for testing, an evaluation suite,
and open-source baseline solutions based on popular LLMs to showcase the usage
of the dataset and to simplify adoption by other researchers. We publish the
benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace
Hub for all the datasets, and link to the GitHub repository with baselines:
https://huggingface.co/spaces/JetBrains-Research/long-code-arena.
667169e90bc08e519555ed61
Egor Bogomolov
claimed_verified
2024-06-19T20:21:03.441Z
64380bed961bb61e463bf93d
https://cdn-avatars.huggingface.co/v1/production/uploads/64380bed961bb61e463bf93d/zsel0Dzv1yU9O8zAxvCBw.jpeg
Egor Bogomolov
user
egor-bogomolov
667169e90bc08e519555ed62
Aleksandra Eliseeva
claimed_verified
2024-06-19T20:21:05.185Z
64a0539e7b57fab3a5d2905e
https://cdn-avatars.huggingface.co/v1/production/uploads/64a0539e7b57fab3a5d2905e/4gZo0l51m_ztFjPkBPGq0.jpeg
Aleksandra Eliseeva
user
saridormi
667169e90bc08e519555ed63
Timur Galimzyanov
admin_assigned
2024-06-20T07:27:06.923Z
63ccde8380ba2ca4152d62f9
/avatars/082414b9cdaf4803766bce3d536c22b7.svg
Timur Galimzyanov
user
galtimur
667169e90bc08e519555ed64
Evgeniy Glukhov
claimed_verified
2024-06-19T08:30:00.356Z
6440ff39ad24e9b2cfba8575
https://cdn-avatars.huggingface.co/v1/production/uploads/6440ff39ad24e9b2cfba8575/QoQC60vbfd2YzTtPAXpM-.jpeg
Evgeniy Glukhov
user
jenyag
667169e90bc08e519555ed65
Anton Shapkin
667169e90bc08e519555ed66
Maria Tigina
claimed_verified
2024-06-19T20:21:09.603Z
6499640bc385f541a0a5a930
https://cdn-avatars.huggingface.co/v1/production/uploads/6499640bc385f541a0a5a930/manqoehehGEa68x-T30xY.jpeg
Maria Tigina
user
tiginamaria
667169e90bc08e519555ed67
Yaroslav Golubev
claimed_verified
2024-06-19T08:29:58.667Z
65ce682c3f1b48716ebfceae
/avatars/75851fbe310845c37d4b80a8ee40e94d.svg
Yaroslav Golubev
user
Areyde
667169e90bc08e519555ed68
Alexander Kovrigin
claimed_verified
2024-06-19T20:21:07.212Z
63b2e4fb922f26a27e75b763
https://cdn-avatars.huggingface.co/v1/production/uploads/63b2e4fb922f26a27e75b763/o0txunjDifS2LOvsnxDHD.png
Alexander Kovrigin
user
waleko
667169e90bc08e519555ed69
Arie van Deursen
667169e90bc08e519555ed6a
Maliheh Izadi
admin_assigned
2024-06-20T07:40:27.300Z
63dd4b2af37111482526e0e9
/avatars/ffd2a07c61ea6272009df6184cb9dcef.svg
Maliheh Izadi
user
MalihehIzadi
667169e90bc08e519555ed6b
Timofey Bryksin
admin_assigned
2024-06-20T07:40:32.370Z
64666e2663e7e09dd031de37
/avatars/8f003d47c816cd9af0f34e7d2f08375c.svg
Timofey Bryksin
user
timofeybryksin
Long Code Arena: a Set of Benchmarks for Long-Context Code Models
Nowadays, the fields of code and natural language processing are evolving
rapidly. In particular, models become better at processing long context windows
- supported context sizes have increased by orders of magnitude over the last
few years. However, there is a shortage of benchmarks for code processing that
go beyond a single file of context, while the most popular ones are limited to
a single method. With this work, we aim to close this gap by introducing Long
Code Arena, a suite of six benchmarks for code processing tasks that require
project-wide context. These tasks cover different aspects of code processing:
library-based code generation, CI builds repair, project-level code completion,
commit message generation, bug localization, and module summarization. For each
task, we provide a manually verified dataset for testing, an evaluation suite,
and open-source baseline solutions based on popular LLMs to showcase the usage
of the dataset and to simplify adoption by other researchers. We publish the
benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace
Hub for all the datasets, and link to the GitHub repository with baselines:
https://huggingface.co/spaces/JetBrains-Research/long-code-arena.
Long Code Arena introduces six benchmarks for code processing tasks requiring project-wide context, supported by open-source datasets and baseline solutions.
context windows benchmarks code processing library-based code generation CI builds repair project-level code completion commit message generation bug localization module summarization LLMs HuggingFace Spaces HuggingFace Hub GitHub repository