SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning
The recent progress in large language models (LLMs), especially the invention
of chain-of-thoughts (CoT) prompting, makes it possible to solve reasoning
problems. However, even the strongest LLMs are still struggling with more
complicated problems that require non-linear thinking and multi-step reasoning.
In this work, we explore whether LLMs have the ability to recognize their own
errors, without resorting to external resources. In particular, we investigate
whether they can be used to identify individual errors within a step-by-step
reasoning. To this end, we propose a zero-shot verification scheme to recognize
such errors. We then use this verification scheme to improve question-answering
performance, by using it to perform weighted voting on different generated
answers. We test the method on three math datasets-GSM8K, MathQA, and MATH-and
find that it successfully recognizes errors and, in turn, increases final
predictive performance.
64c9bc560986bd6fa25a42fe
Ning Miao
admin_assigned
2023-08-02T08:06:08.373Z
6347fa30e3f2de18545d894c
/avatars/b977fbfe87d2d48e43ab0431ca336b1f.svg
NingMiao
user
NingMiao
64c9bc560986bd6fa25a42ff
Yee Whye Teh
64c9bc560986bd6fa25a4300
Tom Rainforth
claimed_verified
2023-08-02T12:28:11.781Z
64ca27e3a81988d07319e842
/avatars/e0c87517352c8e68ffe7ba8e6acc7999.svg
Tom Rainforth 
user
TomRainforth
SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning
The recent progress in large language models (LLMs), especially the invention
of chain-of-thoughts (CoT) prompting, makes it possible to solve reasoning
problems. However, even the strongest LLMs are still struggling with more
complicated problems that require non-linear thinking and multi-step reasoning.
In this work, we explore whether LLMs have the ability to recognize their own
errors, without resorting to external resources. In particular, we investigate
whether they can be used to identify individual errors within a step-by-step
reasoning. To this end, we propose a zero-shot verification scheme to recognize
such errors. We then use this verification scheme to improve question-answering
performance, by using it to perform weighted voting on different generated
answers. We test the method on three math datasets-GSM8K, MathQA, and MATH-and
find that it successfully recognizes errors and, in turn, increases final
predictive performance.
A zero-shot verification scheme improves LLMs' question-answering performance by identifying and correcting errors in step-by-step reasoning.
large language models (LLMs) chain-of-thoughts (CoT) reasoning problems non-linear thinking multi-step reasoning zero-shot verification scheme weighted voting question-answering GSM8K MathQA MATH