4.2.3 Question Answering
We look at Web Questions (Berant et al., 2013),
Natural Questions (Kwiatkowski et al., 2019) and
TriviaQA (Joshi et al., 2017), the three question an-
swering datasets considered by Brown et al. (2020).
For evaluation, we check whether the ﬁrst 20 words
predicted by a model contain the correct answer
instead of requiring an exact match. For Tool-
former, we disable the question answering tool as

7An exception to this is if the model’s prediction contains
an equation (e.g., “The correct answer is 5+3=8”), in which
case we consider the ﬁrst number after the “=” sign to be its
prediction.

Model
GPT-J
GPT-J + CC
Toolformer (disabled)
Toolformer
OPT (66B)
GPT-3 (175B)

WebQS NQ TriviaQA

18.5
18.4
18.9
26.3
18.6
29.0

12.8
12.2
12.6
17.7
11.4
22.6

43.9
45.6
46.7
48.8
45.7
65.9
Toolformer