title: Long Code Arena: a Set of Benchmarks for Long-Context Code Models
summary: Nowadays, the fields of code and natural language processing are evolving
  rapidly. In particular, models become better at processing long context windows
  - supported context sizes have increased by orders of magnitude over the last
  few years. However, there is a shortage of benchmarks for code processing that
  go beyond a single file of context, while the most popular ones are limited to
  a single method. With this work, we aim to close this gap by introducing Long
  Code Arena, a suite of six benchmarks for code processing tasks that require
  project-wide context. These tasks cover different aspects of code processing:
  library-based code generation, CI builds repair, project-level code completion,
  commit message generation, bug localization, and module summarization. For each
  task, we provide a manually verified dataset for testing, an evaluation suite,
  and open-source baseline solutions based on popular LLMs to showcase the usage
  of the dataset and to simplify adoption by other researchers. We publish the
  benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace
  Hub for all the datasets, and link to the GitHub repository with baselines:
  https://huggingface.co/spaces/JetBrains-Research/long-code-arena.
paper_authors._id: 667169e90bc08e519555ed61
paper_authors.name: Egor Bogomolov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:03.441Z
paper_authors._id: 64380bed961bb61e463bf93d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64380bed961bb61e463bf93d/zsel0Dzv1yU9O8zAxvCBw.jpeg
paper_authors.fullname: Egor Bogomolov
paper_authors.type: user
paper_authors.user: egor-bogomolov
paper_authors._id: 667169e90bc08e519555ed62
paper_authors.name: Aleksandra Eliseeva
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:05.185Z
paper_authors._id: 64a0539e7b57fab3a5d2905e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a0539e7b57fab3a5d2905e/4gZo0l51m_ztFjPkBPGq0.jpeg
paper_authors.fullname: Aleksandra Eliseeva
paper_authors.type: user
paper_authors.user: saridormi
paper_authors._id: 667169e90bc08e519555ed63
paper_authors.name: Timur Galimzyanov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:27:06.923Z
paper_authors._id: 63ccde8380ba2ca4152d62f9
paper_authors.avatarUrl: /avatars/082414b9cdaf4803766bce3d536c22b7.svg
paper_authors.fullname: Timur Galimzyanov
paper_authors.type: user
paper_authors.user: galtimur
paper_authors._id: 667169e90bc08e519555ed64
paper_authors.name: Evgeniy Glukhov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T08:30:00.356Z
paper_authors._id: 6440ff39ad24e9b2cfba8575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6440ff39ad24e9b2cfba8575/QoQC60vbfd2YzTtPAXpM-.jpeg
paper_authors.fullname: Evgeniy Glukhov
paper_authors.type: user
paper_authors.user: jenyag
paper_authors._id: 667169e90bc08e519555ed65
paper_authors.name: Anton Shapkin
paper_authors._id: 667169e90bc08e519555ed66
paper_authors.name: Maria Tigina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:09.603Z
paper_authors._id: 6499640bc385f541a0a5a930
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6499640bc385f541a0a5a930/manqoehehGEa68x-T30xY.jpeg
paper_authors.fullname: Maria Tigina
paper_authors.type: user
paper_authors.user: tiginamaria
paper_authors._id: 667169e90bc08e519555ed67
paper_authors.name: Yaroslav Golubev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T08:29:58.667Z
paper_authors._id: 65ce682c3f1b48716ebfceae
paper_authors.avatarUrl: /avatars/75851fbe310845c37d4b80a8ee40e94d.svg
paper_authors.fullname: Yaroslav Golubev
paper_authors.type: user
paper_authors.user: Areyde
paper_authors._id: 667169e90bc08e519555ed68
paper_authors.name: Alexander Kovrigin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:07.212Z
paper_authors._id: 63b2e4fb922f26a27e75b763
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b2e4fb922f26a27e75b763/o0txunjDifS2LOvsnxDHD.png
paper_authors.fullname: Alexander Kovrigin
paper_authors.type: user
paper_authors.user: waleko
paper_authors._id: 667169e90bc08e519555ed69
paper_authors.name: Arie van Deursen
paper_authors._id: 667169e90bc08e519555ed6a
paper_authors.name: Maliheh Izadi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:40:27.300Z
paper_authors._id: 63dd4b2af37111482526e0e9
paper_authors.avatarUrl: /avatars/ffd2a07c61ea6272009df6184cb9dcef.svg
paper_authors.fullname: Maliheh Izadi
paper_authors.type: user
paper_authors.user: MalihehIzadi
paper_authors._id: 667169e90bc08e519555ed6b
paper_authors.name: Timofey Bryksin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T07:40:32.370Z
paper_authors._id: 64666e2663e7e09dd031de37
paper_authors.avatarUrl: /avatars/8f003d47c816cd9af0f34e7d2f08375c.svg
paper_authors.fullname: Timofey Bryksin
paper_authors.type: user
paper_authors.user: timofeybryksin
paper_title: Long Code Arena: a Set of Benchmarks for Long-Context Code Models
paper_summary: Nowadays, the fields of code and natural language processing are evolving
  rapidly. In particular, models become better at processing long context windows
  - supported context sizes have increased by orders of magnitude over the last
  few years. However, there is a shortage of benchmarks for code processing that
  go beyond a single file of context, while the most popular ones are limited to
  a single method. With this work, we aim to close this gap by introducing Long
  Code Arena, a suite of six benchmarks for code processing tasks that require
  project-wide context. These tasks cover different aspects of code processing:
  library-based code generation, CI builds repair, project-level code completion,
  commit message generation, bug localization, and module summarization. For each
  task, we provide a manually verified dataset for testing, an evaluation suite,
  and open-source baseline solutions based on popular LLMs to showcase the usage
  of the dataset and to simplify adoption by other researchers. We publish the
  benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace
  Hub for all the datasets, and link to the GitHub repository with baselines:
  https://huggingface.co/spaces/JetBrains-Research/long-code-arena.
paper_ai_summary: Long Code Arena introduces six benchmarks for code processing tasks requiring project-wide context, supported by open-source datasets and baseline solutions.
paper_ai_keywords: context windows benchmarks code processing library-based code generation CI builds repair project-level code completion commit message generation bug localization module summarization LLMs HuggingFace Spaces HuggingFace Hub GitHub repository