title: Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with
    Code-based Self-Verification
summary: Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has
  brought significant advancements in addressing math reasoning problems. In
  particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter,
  shows remarkable performance on challenging math datasets. In this paper, we
  explore the effect of code on enhancing LLMs' reasoning capability by
  introducing different constraints on the Code Usage Frequency of GPT-4
  Code Interpreter. We found that its success can be largely attributed to its
  powerful skills in generating and executing code, evaluating the output of code
  execution, and rectifying its solution when receiving unreasonable outputs.
  Based on this insight, we propose a novel and effective prompting method,
  explicit code-based self-verification~(CSV), to further
  boost the mathematical reasoning potential of GPT-4 Code Interpreter. This
  method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to
  use code to self-verify its answers. In instances where the verification state
  registers as ``False'', the model shall automatically amend its solution,
  analogous to our approach of rectifying errors during a mathematics
  examination. Furthermore, we recognize that the states of the verification
  result indicate the confidence of a solution, which can improve the
  effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we
  achieve an impressive zero-shot accuracy on MATH dataset (53.9\% to
  84.3\%).
paper_authors._id: 64dc67ba3ae9693d991daa75
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T10:06:07.105Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 64dc67ba3ae9693d991daa76
paper_authors.name: Ke Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:46:47.735Z
paper_authors._id: 64d592c28767727dffa1f002
paper_authors.avatarUrl: /avatars/fe38bcac944a2742dc12c624e62d24ef.svg
paper_authors.fullname: WangKe
paper_authors.type: user
paper_authors.user: scikkk
paper_authors._id: 64dc67ba3ae9693d991daa77
paper_authors.name: Zimu Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-17T13:53:40.565Z
paper_authors._id: 64b0bfef2f2f9c345b87e673
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
paper_authors.fullname: Zimu Lu
paper_authors.type: user
paper_authors.user: luzimu
paper_authors._id: 64dc67ba3ae9693d991daa78
paper_authors.name: Weikang Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-17T13:53:36.022Z
paper_authors._id: 63e3468188181ba2ab0be066
paper_authors.avatarUrl: /avatars/a803d1891adeda86bd31cccde5fcef1e.svg
paper_authors.fullname: Shi Weikang
paper_authors.type: user
paper_authors.user: swk20
paper_authors._id: 64dc67ba3ae9693d991daa79
paper_authors.name: Sichun Luo
paper_authors._id: 64dc67ba3ae9693d991daa7a
paper_authors.name: Zipeng Qin
paper_authors._id: 64dc67ba3ae9693d991daa7b
paper_authors.name: Shaoqing Lu
paper_authors._id: 64dc67ba3ae9693d991daa7c
paper_authors.name: Anya Jia
paper_authors._id: 64dc67ba3ae9693d991daa7d
paper_authors.name: Linqi Song
paper_authors._id: 64dc67ba3ae9693d991daa7e
paper_authors.name: Mingjie Zhan
paper_authors._id: 64dc67ba3ae9693d991daa7f
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:50:36.332Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with
    Code-based Self-Verification
paper_summary: Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has
  brought significant advancements in addressing math reasoning problems. In
  particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter,
  shows remarkable performance on challenging math datasets. In this paper, we
  explore the effect of code on enhancing LLMs' reasoning capability by
  introducing different constraints on the Code Usage Frequency of GPT-4
  Code Interpreter. We found that its success can be largely attributed to its
  powerful skills in generating and executing code, evaluating the output of code
  execution, and rectifying its solution when receiving unreasonable outputs.
  Based on this insight, we propose a novel and effective prompting method,
  explicit code-based self-verification~(CSV), to further
  boost the mathematical reasoning potential of GPT-4 Code Interpreter. This
  method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to
  use code to self-verify its answers. In instances where the verification state
  registers as ``False'', the model shall automatically amend its solution,
  analogous to our approach of rectifying errors during a mathematics
  examination. Furthermore, we recognize that the states of the verification
  result indicate the confidence of a solution, which can improve the
  effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we
  achieve an impressive zero-shot accuracy on MATH dataset (53.9\% to
  84.3\%).
paper_ai_keywords: large language models (LLMs) GPT-4 PaLM-2 math reasoning problems GPT-4 Code Interpreter Code Usage Frequency code generation code execution output evaluation solution rectification code-based self-verification (CSV) zero-shot prompt majority voting MATH dataset