Towards Translating Real-World Code with LLMs: A Study of Translating to Rust
<|reference_start|>Towards Translating Real-World Code with LLMs: A Study of Translating to Rust: Large language models (LLMs) show promise in code translation - the task of translating code written in one programming language to another language - due to their ability to write code in most programming languages. However, LLM's effectiveness on translating real-world code remains largely unstudied. In this work, we perform the first substantial study on LLM-based translation to Rust by assessing the ability of five state-of-the-art LLMs, GPT4, Claude 3, Claude 2.1, Gemini Pro, and Mixtral. We conduct our study on code extracted from real-world open source projects. To enable our study, we develop FLOURINE, an end-to-end code translation tool that uses differential fuzzing to check if a Rust translation is I/O equivalent to the original source program, eliminating the need for pre-existing test cases. As part of our investigation, we assess both the LLM's ability to produce an initially successful translation, as well as their capacity to fix a previously generated buggy one. If the original and the translated programs are not I/O equivalent, we apply a set of automated feedback strategies, including feedback to the LLM with counterexamples. Our results show that the most successful LLM can translate 47% of our benchmarks, and also provides insights into next steps for improvements.<|reference_end|>
@article{eniser2024towards,
    title={Towards Translating Real-World Code with LLMs: A Study of Translating to
  Rust},
    author={Hasan Ferit Eniser, Hanliang Zhang, Cristina David, Meng Wang, Maria
  Christakis, Brandon Paulsen, Joey Dodds, Daniel Kroening},
    journal={arXiv preprint arXiv:2405.11514},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.11514},
    primaryClass={cs.SE}
}