Table 3: Standard prompting versus chain of thought prompting on the four subsets of the MAWPS
benchmark. The point of stratifying the MAWPS benchmark is to show that performance gains are
minimal on easy one-step or two-step problems where large language models already achieve high
performance (e.g., SingleOp, SingleEq, and AddSub).
SingleEq

SingleOp

AddSub

Model
UL2
LaMDA 420M

20B

2B
8B
68B
137B
350M
1.3B
6.7B
175B
-

GPT

Codex
PaLM 8B
62B
540B

MultiArith
standard CoT standard CoT standard CoT standard CoT
5.0 10.7
5.8
1.5
1.8
5.8
5.2
2.4
8.7 32.4
7.6 44.9
0.8
2.3
0.5
2.2
4.5
2.8
33.8 91.7
44.0 96.2
4.2 15.8
7.3 73.7
42.2 94.7

18.0 20.2
2.4
0.4
3.3
2.4
4.5
4.4
23.9 26.0
48.8 58.7
0.2
2.0
1.6
2.4
8.7
4.9
82.7 86.6
86.8 93.1
29.5 28.2
77.2 83.5
86.5 92.3

24.9 27.2
2.8
1.0
4.1
4.6
8.0
7.0
36.5 40.8
73.2 76.2
1.8
3.2
3.0
5.3
13.5
3.9
90.9 88.8
93.1 91.8
41.8 46.6
87.9 85.6
94.1 94.1
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models