title: SURGE: On the Potential of Large Language Models as General-Purpose
    Surrogate Code Executors
summary: Large language models (LLMs) have demonstrated remarkable capabilities in
  code-related tasks, such as code understanding and code generation. However, an
  equally important yet underexplored question is whether LLMs can serve as
  general-purpose surrogate code executors, to predict the output and behavior of
  a program without actually running it. To systematically investigate this
  capability, we introduce SURGE, a comprehensive benchmark covering eight key
  aspects: multi-language programming tasks, competition-level programming
  problems, repository-level code analysis, high-cost scientific computing,
  time-complexity-intensive algorithms, buggy code analysis, programs dependent
  on specific compilers or execution environments, and formal mathematical proof
  verification. We evaluate multiple open-source and proprietary LLMs on SURGE
  and conduct a scaling study to analyze the impact of model size and training
  data scale on surrogate execution accuracy. Additionally, we categorize model
  prediction errors and explore potential areas for improvement. Our findings
  indicate that while LLMs can predict code execution results in certain cases,
  they exhibit limitations in general-purpose surrogate execution. This study
  provides empirical insights into the feasibility of using LLMs as surrogate
  code executors. Code and dataset are released at
  https://github.com/Imbernoulli/SURGE.
paper_authors._id: 67b4221bbc387d2eda6f8637
paper_authors.name: Bohan Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:31:06.388Z
paper_authors._id: 650267e7e751d03da933a24a
paper_authors.avatarUrl: /avatars/f047a047d1de304cd97027463541bdf3.svg
paper_authors.fullname: Bohan22
paper_authors.type: user
paper_authors.user: Bohan22
paper_authors._id: 67b4221bbc387d2eda6f8638
paper_authors.name: Siqiao Huang
paper_authors._id: 67b4221bbc387d2eda6f8639
paper_authors.name: Zichen Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:31:08.469Z
paper_authors._id: 67286718746a95c09d04cb1d
paper_authors.avatarUrl: /avatars/317efa8459cca08c2ff56c3ab116e15c.svg
paper_authors.fullname: Zichen Liang
paper_authors.type: user
paper_authors.user: zcliang22
paper_title: SURGE: On the Potential of Large Language Models as General-Purpose
    Surrogate Code Executors
paper_summary: Large language models (LLMs) have demonstrated remarkable capabilities in
  code-related tasks, such as code understanding and code generation. However, an
  equally important yet underexplored question is whether LLMs can serve as
  general-purpose surrogate code executors, to predict the output and behavior of
  a program without actually running it. To systematically investigate this
  capability, we introduce SURGE, a comprehensive benchmark covering eight key
  aspects: multi-language programming tasks, competition-level programming
  problems, repository-level code analysis, high-cost scientific computing,
  time-complexity-intensive algorithms, buggy code analysis, programs dependent
  on specific compilers or execution environments, and formal mathematical proof
  verification. We evaluate multiple open-source and proprietary LLMs on SURGE
  and conduct a scaling study to analyze the impact of model size and training
  data scale on surrogate execution accuracy. Additionally, we categorize model
  prediction errors and explore potential areas for improvement. Our findings
  indicate that while LLMs can predict code execution results in certain cases,
  they exhibit limitations in general-purpose surrogate execution. This study
  provides empirical insights into the feasibility of using LLMs as surrogate
  code executors. Code and dataset are released at
  https://github.com/Imbernoulli/SURGE.