title: Code Execution with Pre-trained Language Models
summary: Code execution is a fundamental aspect of programming language semantics that
  reflects the exact behavior of the code. However, most pre-trained models for
  code intelligence ignore the execution trace and only rely on source code and
  syntactic structures. In this paper, we investigate how well pre-trained models
  can understand and perform code execution. We develop a mutation-based data
  augmentation technique to create a large-scale and realistic Python dataset and
  task for code execution, which challenges existing models such as Codex. We
  then present CodeExecutor, a Transformer model that leverages code execution
  pre-training and curriculum learning to enhance its semantic comprehension. We
  evaluate CodeExecutor on code execution and show its promising performance and
  limitations. We also demonstrate its potential benefits for code intelligence
  tasks such as zero-shot code-to-code search and text-to-code generation. Our
  analysis provides insights into the learning and generalization abilities of
  pre-trained models for code execution.
paper_authors._id: 645af8c4e7e8a4b57bdc4f50
paper_authors.name: Chenxiao Liu
paper_authors._id: 645af8c4e7e8a4b57bdc4f51
paper_authors.name: Shuai Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:24:17.666Z
paper_authors._id: 617a3066f5910ea887210d91
paper_authors.avatarUrl: /avatars/5193c5f523459f3d977defb257ef6f0d.svg
paper_authors.fullname: Shuai Lu
paper_authors.type: user
paper_authors.user: shuailu
paper_authors._id: 645af8c4e7e8a4b57bdc4f52
paper_authors.name: Weizhu Chen
paper_authors._id: 645af8c4e7e8a4b57bdc4f53
paper_authors.name: Daxin Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:25:24.554Z
paper_authors._id: 60d4440fe648443279aaffd8
paper_authors.avatarUrl: /avatars/bf7209c1f14ae120f5bfda5fda1301b7.svg
paper_authors.fullname: Daxin Jiang
paper_authors.type: user
paper_authors.user: djiang
paper_authors._id: 645af8c4e7e8a4b57bdc4f54
paper_authors.name: Alexey Svyatkovskiy
paper_authors._id: 645af8c4e7e8a4b57bdc4f55
paper_authors.name: Shengyu Fu
paper_authors._id: 645af8c4e7e8a4b57bdc4f56
paper_authors.name: Neel Sundaresan
paper_authors._id: 645af8c4e7e8a4b57bdc4f57
paper_authors.name: Nan Duan
paper_title: Code Execution with Pre-trained Language Models
paper_summary: Code execution is a fundamental aspect of programming language semantics that
  reflects the exact behavior of the code. However, most pre-trained models for
  code intelligence ignore the execution trace and only rely on source code and
  syntactic structures. In this paper, we investigate how well pre-trained models
  can understand and perform code execution. We develop a mutation-based data
  augmentation technique to create a large-scale and realistic Python dataset and
  task for code execution, which challenges existing models such as Codex. We
  then present CodeExecutor, a Transformer model that leverages code execution
  pre-training and curriculum learning to enhance its semantic comprehension. We
  evaluate CodeExecutor on code execution and show its promising performance and
  limitations. We also demonstrate its potential benefits for code intelligence
  tasks such as zero-shot code-to-code search and text-to-code generation. Our
  analysis provides insights into the learning and generalization abilities of
  pre-trained models for code execution.
paper_ai_keywords: transformer code execution pre-training curriculum learning zero-shot code-to-code search text-to-code generation