MMLU (Hendrycks et al., 2020) is a massive multitask language understanding benchmark, covering
multiple-choice questions in 57 knowledge domains, including the humanities, STEM, and social
sciences. CoQA (Reddy et al., 2019) is a large-scale dataset for Conversational Question Answering
systems, measuring the model’s ability to process a text passage and answer a series of interconnected
questions. We compare StarCoderBase and StarCoder with CodeGen-16B-Multi (Nijkamp et al.,
2023), GPT-NeoX (Black et al., 2022), LLaMA-7B, and LLaMA-13B (Touvron et al., 2023).
We present the 5-shot accuracy for MMLU in Table 22, and the zero-shot F1 scores for CoQA in Table
23. On MMLU, StarCoderBase outperforms CodeGen-16B-Multi significantly (34.2% to 27.8%),
and even outperforms GPT-NeoX by a small margin (32.9%). Nevertheless, both LLaMA models
outperform StarCoderBase. On CoQA, StarCoderBase performs better than CodeGen-16B-Multi but
is outperformed by LLaMA and GPT-NeoX.

25
StarCoder_paper (1)