MIGRATION-BENCH: Repository-Level Code Migration Benchmark from Java 8
With the rapid advancement of powerful large language models (LLMs) in recent
years, a wide range of software engineering tasks can now be addressed using
LLMs, significantly enhancing productivity and scalability. Numerous benchmark
datasets have been developed to evaluate the coding capabilities of these
models, while they primarily focus on problem-solving and issue-resolution
tasks. In contrast, we introduce a new coding benchmark MIGRATION-BENCH with a
distinct focus: code migration. MIGRATION-BENCH aims to serve as a
comprehensive benchmark for migration from Java 8 to the latest long-term
support (LTS) versions (Java 17, 21), MIGRATION-BENCH includes a full dataset
and its subset selected with 5,102 and 300 repositories respectively.
Selected is a representative subset curated for complexity and difficulty,
offering a versatile resource to support research in the field of code
migration. Additionally, we provide a comprehensive evaluation framework to
facilitate rigorous and standardized assessment of LLMs on this challenging
task. We further propose SD-Feedback and demonstrate that LLMs can effectively
tackle repository-level code migration to Java 17. For the selected subset with
Claude-3.5-Sonnet-v2, SD-Feedback achieves 62.33% and 27.00% success rate
(pass@1) for minimal and maximal migration respectively. The benchmark dataset
and source code are available at:
https://huggingface.co/collections/AmazonScience and
https://github.com/amazon-science/self_debug respectively.
682563f807f74666ec373332
Linbo Liu
admin_assigned
2025-05-21T10:28:01.555Z
682573f5a8792137ebca0a70
/avatars/358767039db06c41c2bcee55a081cb21.svg
Linbo Liu
user
linboliu
682563f807f74666ec373333
Xinle Liu
claimed_verified
2025-05-15T10:30:40.773Z
636c32ae181c81c337f086b9
https://cdn-avatars.huggingface.co/v1/production/uploads/636c32ae181c81c337f086b9/9zHTHmwzSeLMJWuwHCqAe.jpeg
Xinle Sheila Liu
user
sliuxl
682563f807f74666ec373334
Qiang Zhou
682563f807f74666ec373335
Lin Chen
682563f807f74666ec373336
Yihan Liu
682563f807f74666ec373337
Hoan Nguyen
682563f807f74666ec373338
Behrooz Omidvar-Tehrani
admin_assigned
2025-05-21T10:28:28.110Z
63aa8f1b8780edbabb18f9ad
/avatars/a184f9816bedf5f9f49975f3334fd24a.svg
Behrooz Omidvar-Tehrani
user
omidvarb
682563f807f74666ec373339
Xi Shen
682563f807f74666ec37333a
Jun Huan
682563f807f74666ec37333b
Omer Tripp
682563f807f74666ec37333c
Anoop Deoras
MIGRATION-BENCH: Repository-Level Code Migration Benchmark from Java 8
With the rapid advancement of powerful large language models (LLMs) in recent
years, a wide range of software engineering tasks can now be addressed using
LLMs, significantly enhancing productivity and scalability. Numerous benchmark
datasets have been developed to evaluate the coding capabilities of these
models, while they primarily focus on problem-solving and issue-resolution
tasks. In contrast, we introduce a new coding benchmark MIGRATION-BENCH with a
distinct focus: code migration. MIGRATION-BENCH aims to serve as a
comprehensive benchmark for migration from Java 8 to the latest long-term
support (LTS) versions (Java 17, 21), MIGRATION-BENCH includes a full dataset
and its subset selected with 5,102 and 300 repositories respectively.
Selected is a representative subset curated for complexity and difficulty,
offering a versatile resource to support research in the field of code
migration. Additionally, we provide a comprehensive evaluation framework to
facilitate rigorous and standardized assessment of LLMs on this challenging
task. We further propose SD-Feedback and demonstrate that LLMs can effectively
tackle repository-level code migration to Java 17. For the selected subset with
Claude-3.5-Sonnet-v2, SD-Feedback achieves 62.33% and 27.00% success rate
(pass@1) for minimal and maximal migration respectively. The benchmark dataset
and source code are available at:
https://huggingface.co/collections/AmazonScience and
https://github.com/amazon-science/self_debug respectively.
A new benchmark MIGRATION-BENCH evaluates large language models on Java code migration tasks, providing a dataset and framework for repository-level migration assessment.
large language models code migration MIGRATION-BENCH Java 8 Java 17 Java 21 repository-level migration SD-Feedback