title: Chain of Code: Reasoning with a Language Model-Augmented Code Emulator
summary: Code provides a general syntactic structure to build complex programs and
  perform precise computations when paired with a code interpreter -- we
  hypothesize that language models (LMs) can leverage code-writing to improve
  Chain of Thought reasoning not only for logic and arithmetic tasks, but also
  for linguistic ones (and in particular, those that are a mix of both). For
  example, consider prompting an LM to write code that counts the number of times
  it detects sarcasm in an essay: the LM may struggle to write an implementation
  for "detect_sarcasm(string)" that can be executed by the interpreter (handling
  the edge cases would be insurmountable). However, LMs may still produce a valid
  solution if they are used not only to write the code, but also to selectively
  "emulate" the interpreter by generating the expected output of
  "detect_sarcasm(string)" and other lines of code (e.g., that the interpreter
  could not compile). In this work, we propose Chain of Code (CoT), a simple yet
  surprisingly effective extension that improves LM code-driven reasoning. The
  key idea is to encourage LMs to format linguistic sub-tasks in a program as
  flexible pseudocode that the compiler can explicitly catch undefined behaviors
  and hand off to simulate with an LM (as an "LMulator"). Experiments demonstrate
  that Chain of Code outperforms Chain of Thought and other baselines across a
  variety of benchmarks; on BIG-Bench Hard, Chain of Code achieves 84%, a gain of
  12% over Chain of Thought. CoT scales well with large and small models alike,
  and broadens the scope of reasoning questions that LMs can correctly answer by
  "thinking in code". Project webpage: https://chain-of-code.github.io/.
paper_authors._id: 65727dbec2913e69b4c5c215
paper_authors.name: Chengshu Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-22T00:20:40.845Z
paper_authors._id: 664d3a3079f09472c1193e06
paper_authors.avatarUrl: /avatars/3cf4591f36230137bb059e7f2de8bbc3.svg
paper_authors.fullname: ChengshuLi
paper_authors.type: user
paper_authors.user: chengshu
paper_authors._id: 65727dbec2913e69b4c5c216
paper_authors.name: Jacky Liang
paper_authors._id: 65727dbec2913e69b4c5c217
paper_authors.name: Andy Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:59:43.557Z
paper_authors._id: 62d5f9d705461e9459d20b91
paper_authors.avatarUrl: /avatars/120917da2d8587674505163fb960e27d.svg
paper_authors.fullname: Andy Zeng
paper_authors.type: user
paper_authors.user: andyzeng
paper_authors._id: 65727dbec2913e69b4c5c218
paper_authors.name: Xinyun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:02:55.074Z
paper_authors._id: 64d0268001931c60161e026a
paper_authors.avatarUrl: /avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
paper_authors.fullname: Xinyun Chen
paper_authors.type: user
paper_authors.user: xinyunchen
paper_authors._id: 65727dbec2913e69b4c5c219
paper_authors.name: Karol Hausman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:03:03.055Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 65727dbec2913e69b4c5c21a
paper_authors.name: Dorsa Sadigh
paper_authors._id: 65727dbec2913e69b4c5c21b
paper_authors.name: Sergey Levine
paper_authors._id: 65727dbec2913e69b4c5c21c
paper_authors.name: Li Fei-Fei
paper_authors._id: 65727dbec2913e69b4c5c21d
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-08T08:00:20.825Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 65727dbec2913e69b4c5c21e
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:03:29.078Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_title: Chain of Code: Reasoning with a Language Model-Augmented Code Emulator
paper_summary: Code provides a general syntactic structure to build complex programs and
  perform precise computations when paired with a code interpreter -- we
  hypothesize that language models (LMs) can leverage code-writing to improve
  Chain of Thought reasoning not only for logic and arithmetic tasks, but also
  for linguistic ones (and in particular, those that are a mix of both). For
  example, consider prompting an LM to write code that counts the number of times
  it detects sarcasm in an essay: the LM may struggle to write an implementation
  for "detect_sarcasm(string)" that can be executed by the interpreter (handling
  the edge cases would be insurmountable). However, LMs may still produce a valid
  solution if they are used not only to write the code, but also to selectively
  "emulate" the interpreter by generating the expected output of
  "detect_sarcasm(string)" and other lines of code (e.g., that the interpreter
  could not compile). In this work, we propose Chain of Code (CoT), a simple yet
  surprisingly effective extension that improves LM code-driven reasoning. The
  key idea is to encourage LMs to format linguistic sub-tasks in a program as
  flexible pseudocode that the compiler can explicitly catch undefined behaviors
  and hand off to simulate with an LM (as an "LMulator"). Experiments demonstrate
  that Chain of Code outperforms Chain of Thought and other baselines across a
  variety of benchmarks; on BIG-Bench Hard, Chain of Code achieves 84%, a gain of
  12% over Chain of Thought. CoT scales well with large and small models alike,
  and broadens the scope of reasoning questions that LMs can correctly answer by
  "thinking in code". Project webpage: https://chain-of-code.github.io/.
paper_ai_keywords: language models (LMs) code-writing Chain of Thought reasoning synthesis computational requirements interpreter code "detect_sarcasm(string)" Con insurmountable Chain of Code (CoT) pseudocode undefined behaviors LMulator BIG-Bench Hard