3.5 Code generation
We evaluate the ability of our models to write
code from a natural language description on two
benchmarks: HumanEval (Chen et al., 2021) and
MBPP (Austin et al., 2021). For both tasks, the
model receives a description of the program in a
few sentences, as well as a few input-output ex-
amples. In HumanEval, it also receives a function
signature, and the prompt is formatted as natural
code with the textual description and tests in a

MATH +maj1@k GSM8k +maj1@k

Params HumanEval

MBPP

PaLM

Minerva

LLaMA

8B 1.5
62B 4.4
540B 8.8
8B 14.1
62B 27.6
540B 33.6
7B 2.9
13B 3.9
33B 7.1
65B 10.6

-
-
-

25.4
43.4
50.3
6.9
8.8
15.2
20.5

4.1
33.0
56.5
16.2
52.4
68.5
11.0
17.8
35.6
50.9

-
-
-

28.4
68.5
78.5
18.1
29.3
53.1
69.7
LLaMA- Open and Efficient Foundation Language Models