CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation
C-to-Rust transpilation is essential for modernizing legacy C code while
enhancing safety and interoperability with modern Rust ecosystems. However, no
dataset currently exists for evaluating whether a system can transpile C into
safe Rust that passes a set of test cases. We introduce CRUST-Bench, a dataset
of 100 C repositories, each paired with manually-written interfaces in safe
Rust as well as test cases that can be used to validate correctness of the
transpilation. By considering entire repositories rather than isolated
functions, CRUST-Bench captures the challenges of translating complex projects
with dependencies across multiple files. The provided Rust interfaces provide
explicit specifications that ensure adherence to idiomatic, memory-safe Rust
patterns, while the accompanying test cases enforce functional correctness. We
evaluate state-of-the-art large language models (LLMs) on this task and find
that safe and idiomatic Rust generation is still a challenging problem for
various state-of-the-art methods and techniques. We also provide insights into
the errors LLMs usually make in transpiling code from C to safe Rust. The best
performing model, OpenAI o1, is able to solve only 15 tasks in a single-shot
setting. Improvements on CRUST-Bench would lead to improved transpilation
systems that can reason about complex scenarios and help in migrating legacy
codebases from C into languages like Rust that ensure memory safety. You can
find the dataset and code at https://github.com/anirudhkhatry/CRUST-bench.
6807bf3e70a0cec724b8a011
Anirudh Khatry
claimed_verified
2025-04-23T08:30:04.897Z
6697abd4be7ce6de07140e72
/avatars/1598decabd9975d755082c8fb4e5962d.svg
Anirudh Khatry
user
anirudhkhatry
6807bf3e70a0cec724b8a012
Robert Zhang
6807bf3e70a0cec724b8a013
Jia Pan
6807bf3e70a0cec724b8a014
Ziteng Wang
6807bf3e70a0cec724b8a015
Qiaochu Chen
6807bf3e70a0cec724b8a016
Greg Durrett
admin_assigned
2025-04-24T11:56:46.592Z
65be9918b54ab5b37d1b67a7
/avatars/9953707affb6881724c8efb2abf0c668.svg
Greg Durrett
user
gregdurrett
6807bf3e70a0cec724b8a017
Isil Dillig
CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation
C-to-Rust transpilation is essential for modernizing legacy C code while
enhancing safety and interoperability with modern Rust ecosystems. However, no
dataset currently exists for evaluating whether a system can transpile C into
safe Rust that passes a set of test cases. We introduce CRUST-Bench, a dataset
of 100 C repositories, each paired with manually-written interfaces in safe
Rust as well as test cases that can be used to validate correctness of the
transpilation. By considering entire repositories rather than isolated
functions, CRUST-Bench captures the challenges of translating complex projects
with dependencies across multiple files. The provided Rust interfaces provide
explicit specifications that ensure adherence to idiomatic, memory-safe Rust
patterns, while the accompanying test cases enforce functional correctness. We
evaluate state-of-the-art large language models (LLMs) on this task and find
that safe and idiomatic Rust generation is still a challenging problem for
various state-of-the-art methods and techniques. We also provide insights into
the errors LLMs usually make in transpiling code from C to safe Rust. The best
performing model, OpenAI o1, is able to solve only 15 tasks in a single-shot
setting. Improvements on CRUST-Bench would lead to improved transpilation
systems that can reason about complex scenarios and help in migrating legacy
codebases from C into languages like Rust that ensure memory safety. You can
find the dataset and code at https://github.com/anirudhkhatry/CRUST-bench.
A dataset, CRUST-Bench, evaluates the ability of large language models to transpile C to safe Rust, highlighting the challenges in maintaining idiomatic patterns and memory safety.
CRUST-Bench large language models LLMs C-to-Rust transpilation safe Rust idiomatic Rust memory safety dependencies functional correctness single-shot setting