Sensitivity to exemplars is a key consideration of prompt-
ing approaches—for instance, varying the permutation of
few-shot exemplars can cause the accuracy of GPT-3 on
SST-2 to range from near chance (54.3%) to near state of
the art (93.4%) (Zhao et al., 2021). In this ﬁnal subsec-
tion, we evaluate robustness to chains of thought written
by different annotators. In addition to the results above,
which used chains of thought written by an Annotator
A, two other co-authors of this paper (Annotators B and
C) independently wrote chains of thought for the same
few-shot exemplars (shown in Appendix H). Annotator A
also wrote another chain of thought that was more concise
than the original, following the style of solutions given in
Cobbe et al. (2021).1
Figure 6 shows these results for LaMDA 137B on GSM8K
and MAWPS (ablation results for other datasets are given
in Appendix Table 6 / Table 7). Although there is variance
among different chain of thought annotations, as would be
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models