Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with
  Code-based Self-Verification
Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has
brought significant advancements in addressing math reasoning problems. In
particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter,
shows remarkable performance on challenging math datasets. In this paper, we
explore the effect of code on enhancing LLMs' reasoning capability by
introducing different constraints on the Code Usage Frequency of GPT-4
Code Interpreter. We found that its success can be largely attributed to its
powerful skills in generating and executing code, evaluating the output of code
execution, and rectifying its solution when receiving unreasonable outputs.
Based on this insight, we propose a novel and effective prompting method,
explicit code-based self-verification~(CSV), to further
boost the mathematical reasoning potential of GPT-4 Code Interpreter. This
method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to
use code to self-verify its answers. In instances where the verification state
registers as ``False'', the model shall automatically amend its solution,
analogous to our approach of rectifying errors during a mathematics
examination. Furthermore, we recognize that the states of the verification
result indicate the confidence of a solution, which can improve the
effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we
achieve an impressive zero-shot accuracy on MATH dataset (53.9\% to
84.3\%).
64dc67ba3ae9693d991daa75
Aojun Zhou
claimed_verified
2024-03-22T10:06:07.105Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
64dc67ba3ae9693d991daa76
Ke Wang
claimed_verified
2024-10-11T07:46:47.735Z
64d592c28767727dffa1f002
/avatars/fe38bcac944a2742dc12c624e62d24ef.svg
WangKe
user
scikkk
64dc67ba3ae9693d991daa77
Zimu Lu
claimed_verified
2023-08-17T13:53:40.565Z
64b0bfef2f2f9c345b87e673
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
Zimu Lu
user
luzimu
64dc67ba3ae9693d991daa78
Weikang Shi
claimed_verified
2023-08-17T13:53:36.022Z
63e3468188181ba2ab0be066
/avatars/93fa65f09898dc378dc7da737bed4145.svg
Shi Weikang
user
swk20
64dc67ba3ae9693d991daa79
Sichun Luo
64dc67ba3ae9693d991daa7a
Zipeng Qin
64dc67ba3ae9693d991daa7b
Shaoqing Lu
64dc67ba3ae9693d991daa7c
Anya Jia
64dc67ba3ae9693d991daa7d
Linqi Song
64dc67ba3ae9693d991daa7e
Mingjie Zhan
64dc67ba3ae9693d991daa7f
Hongsheng Li
claimed_verified
2024-02-05T10:50:36.332Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with
  Code-based Self-Verification
Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has
brought significant advancements in addressing math reasoning problems. In
particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter,
shows remarkable performance on challenging math datasets. In this paper, we
explore the effect of code on enhancing LLMs' reasoning capability by
introducing different constraints on the Code Usage Frequency of GPT-4
Code Interpreter. We found that its success can be largely attributed to its
powerful skills in generating and executing code, evaluating the output of code
execution, and rectifying its solution when receiving unreasonable outputs.
Based on this insight, we propose a novel and effective prompting method,
explicit code-based self-verification~(CSV), to further
boost the mathematical reasoning potential of GPT-4 Code Interpreter. This
method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to
use code to self-verify its answers. In instances where the verification state
registers as ``False'', the model shall automatically amend its solution,
analogous to our approach of rectifying errors during a mathematics
examination. Furthermore, we recognize that the states of the verification
result indicate the confidence of a solution, which can improve the
effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we
achieve an impressive zero-shot accuracy on MATH dataset (53.9\% to
84.3\%).
A novel code-based self-verification method significantly improves math reasoning in GPT-4 Code Interpreter, enhancing accuracy on the MATH dataset.
GPT-4 GPT-4 Code Interpreter PaLM-2 large language models math reasoning code usage frequency code generation code execution self-verification zero-shot accuracy