SURGE: On the Potential of Large Language Models as General-Purpose
  Surrogate Code Executors
Large language models (LLMs) have demonstrated remarkable capabilities in
code-related tasks, such as code understanding and code generation. However, an
equally important yet underexplored question is whether LLMs can serve as
general-purpose surrogate code executors, to predict the output and behavior of
a program without actually running it. To systematically investigate this
capability, we introduce SURGE, a comprehensive benchmark covering eight key
aspects: multi-language programming tasks, competition-level programming
problems, repository-level code analysis, high-cost scientific computing,
time-complexity-intensive algorithms, buggy code analysis, programs dependent
on specific compilers or execution environments, and formal mathematical proof
verification. We evaluate multiple open-source and proprietary LLMs on SURGE
and conduct a scaling study to analyze the impact of model size and training
data scale on surrogate execution accuracy. Additionally, we categorize model
prediction errors and explore potential areas for improvement. Our findings
indicate that while LLMs can predict code execution results in certain cases,
they exhibit limitations in general-purpose surrogate execution. This study
provides empirical insights into the feasibility of using LLMs as surrogate
code executors. Code and dataset are released at
https://github.com/Imbernoulli/SURGE.
67b4221bbc387d2eda6f8637
Bohan Lyu
claimed_verified
2025-02-18T09:31:06.388Z
650267e7e751d03da933a24a
/avatars/f047a047d1de304cd97027463541bdf3.svg
Bohan22
user
Bohan22
67b4221bbc387d2eda6f8638
Siqiao Huang
admin_assigned
2025-05-21T18:03:56.357Z
66b06fe4ce2224a4d066cc0a
https://cdn-avatars.huggingface.co/v1/production/uploads/66b06fe4ce2224a4d066cc0a/m7O-cdeRQVU1yc6WVmMzW.png
knightnemo
user
knightnemo
67b4221bbc387d2eda6f8639
Zichen Liang
claimed_verified
2025-02-18T09:31:08.469Z
67286718746a95c09d04cb1d
/avatars/317efa8459cca08c2ff56c3ab116e15c.svg
Zichen Liang
user
zcliang22
SURGE: On the Potential of Large Language Models as General-Purpose
  Surrogate Code Executors
Large language models (LLMs) have demonstrated remarkable capabilities in
code-related tasks, such as code understanding and code generation. However, an
equally important yet underexplored question is whether LLMs can serve as
general-purpose surrogate code executors, to predict the output and behavior of
a program without actually running it. To systematically investigate this
capability, we introduce SURGE, a comprehensive benchmark covering eight key
aspects: multi-language programming tasks, competition-level programming
problems, repository-level code analysis, high-cost scientific computing,
time-complexity-intensive algorithms, buggy code analysis, programs dependent
on specific compilers or execution environments, and formal mathematical proof
verification. We evaluate multiple open-source and proprietary LLMs on SURGE
and conduct a scaling study to analyze the impact of model size and training
data scale on surrogate execution accuracy. Additionally, we categorize model
prediction errors and explore potential areas for improvement. Our findings
indicate that while LLMs can predict code execution results in certain cases,
they exhibit limitations in general-purpose surrogate execution. This study
provides empirical insights into the feasibility of using LLMs as surrogate
code executors. Code and dataset are released at
https://github.com/Imbernoulli/SURGE.
LLMs evaluated on SURGE demonstrate limited capabilities as general-purpose surrogate code executors across various programming tasks.
large language models LLMs surrogate code executors SURGE benchmark multi-language programming competition-level problems repository-level code analysis high-cost scientific computing time-complexity-intensive algorithms buggy code analysis specific compilers execution environments formal mathematical proof verification model prediction errors