Recent work has shown that Code LLMs can be effective arithmetic and symbolic reasoners by
using a technique called Program-Aided Language models (PAL; Gao et al., 2022). With PAL, the
LLM reads the reasoning problem and generates Python programs as the intermediate reasoning
steps, which are then executed by the Python interpreter to produce the answer. In contrast, the
Chain-of-Thought method (CoT; Wei et al., 2022b) prompts the LLM to produce the reasoning steps
in natural language before generating the answer.
We investigate the reasoning capabilities of StarCoderBase on GSM8K (Cobbe et al., 2021), a set of
middle-school math word problems. We compare with the two CodeGen-16B models (Nijkamp et al.,
2023) and the family of LLaMA models (Touvron et al., 2023). The results of our evaluation are
presented in Table 21, where we provide both CoT and PAL results for StarCoderBase and LLaMA.
In line with previous results comparing PAL to CoT on Code LLMs (Gao et al., 2022), we find
StarCoder_paper (1)