title: MIGRATION-BENCH: Repository-Level Code Migration Benchmark from Java 8
summary: With the rapid advancement of powerful large language models (LLMs) in recent
  years, a wide range of software engineering tasks can now be addressed using
  LLMs, significantly enhancing productivity and scalability. Numerous benchmark
  datasets have been developed to evaluate the coding capabilities of these
  models, while they primarily focus on problem-solving and issue-resolution
  tasks. In contrast, we introduce a new coding benchmark MIGRATION-BENCH with a
  distinct focus: code migration. MIGRATION-BENCH aims to serve as a
  comprehensive benchmark for migration from Java 8 to the latest long-term
  support (LTS) versions (Java 17, 21), MIGRATION-BENCH includes a full dataset
  and its subset selected with 5,102 and 300 repositories respectively.
  Selected is a representative subset curated for complexity and difficulty,
  offering a versatile resource to support research in the field of code
  migration. Additionally, we provide a comprehensive evaluation framework to
  facilitate rigorous and standardized assessment of LLMs on this challenging
  task. We further propose SD-Feedback and demonstrate that LLMs can effectively
  tackle repository-level code migration to Java 17. For the selected subset with
  Claude-3.5-Sonnet-v2, SD-Feedback achieves 62.33% and 27.00% success rate
  (pass@1) for minimal and maximal migration respectively. The benchmark dataset
  and source code are available at:
  https://huggingface.co/collections/AmazonScience and
  https://github.com/amazon-science/self_debug respectively.
paper_authors._id: 682563f807f74666ec373332
paper_authors.name: Linbo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:28:01.555Z
paper_authors._id: 682573f5a8792137ebca0a70
paper_authors.avatarUrl: /avatars/358767039db06c41c2bcee55a081cb21.svg
paper_authors.fullname: Linbo Liu
paper_authors.type: user
paper_authors.user: linboliu
paper_authors._id: 682563f807f74666ec373333
paper_authors.name: Xinle Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:30:40.773Z
paper_authors._id: 636c32ae181c81c337f086b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636c32ae181c81c337f086b9/9zHTHmwzSeLMJWuwHCqAe.jpeg
paper_authors.fullname: Xinle Sheila Liu
paper_authors.type: user
paper_authors.user: sliuxl
paper_authors._id: 682563f807f74666ec373334
paper_authors.name: Qiang Zhou
paper_authors._id: 682563f807f74666ec373335
paper_authors.name: Lin Chen
paper_authors._id: 682563f807f74666ec373336
paper_authors.name: Yihan Liu
paper_authors._id: 682563f807f74666ec373337
paper_authors.name: Hoan Nguyen
paper_authors._id: 682563f807f74666ec373338
paper_authors.name: Behrooz Omidvar-Tehrani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:28:28.110Z
paper_authors._id: 63aa8f1b8780edbabb18f9ad
paper_authors.avatarUrl: /avatars/a184f9816bedf5f9f49975f3334fd24a.svg
paper_authors.fullname: Behrooz Omidvar-Tehrani
paper_authors.type: user
paper_authors.user: omidvarb
paper_authors._id: 682563f807f74666ec373339
paper_authors.name: Xi Shen
paper_authors._id: 682563f807f74666ec37333a
paper_authors.name: Jun Huan
paper_authors._id: 682563f807f74666ec37333b
paper_authors.name: Omer Tripp
paper_authors._id: 682563f807f74666ec37333c
paper_authors.name: Anoop Deoras
paper_title: MIGRATION-BENCH: Repository-Level Code Migration Benchmark from Java 8
paper_summary: With the rapid advancement of powerful large language models (LLMs) in recent
  years, a wide range of software engineering tasks can now be addressed using
  LLMs, significantly enhancing productivity and scalability. Numerous benchmark
  datasets have been developed to evaluate the coding capabilities of these
  models, while they primarily focus on problem-solving and issue-resolution
  tasks. In contrast, we introduce a new coding benchmark MIGRATION-BENCH with a
  distinct focus: code migration. MIGRATION-BENCH aims to serve as a
  comprehensive benchmark for migration from Java 8 to the latest long-term
  support (LTS) versions (Java 17, 21), MIGRATION-BENCH includes a full dataset
  and its subset selected with 5,102 and 300 repositories respectively.
  Selected is a representative subset curated for complexity and difficulty,
  offering a versatile resource to support research in the field of code
  migration. Additionally, we provide a comprehensive evaluation framework to
  facilitate rigorous and standardized assessment of LLMs on this challenging
  task. We further propose SD-Feedback and demonstrate that LLMs can effectively
  tackle repository-level code migration to Java 17. For the selected subset with
  Claude-3.5-Sonnet-v2, SD-Feedback achieves 62.33% and 27.00% success rate
  (pass@1) for minimal and maximal migration respectively. The benchmark dataset
  and source code are available at:
  https://huggingface.co/collections/AmazonScience and
  https://github.com/amazon-science/self_debug respectively.
paper_ai_summary: A new benchmark MIGRATION-BENCH evaluates large language models on Java code migration tasks, providing a dataset and framework for repository-level migration assessment.
paper_ai_keywords: large language models code migration MIGRATION-BENCH Java 8 Java 17 Java 21 repository-level migration SD-Feedback