4.2.5 Temporal Datasets
To investigate the calendar API’s utility, we eval-
uate all models on TEMPLAMA (Dhingra et al.,
2022) and a new dataset that we call DATESET.
TEMPLAMA is a dataset built from Wikidata that
contains cloze queries about facts that change with
time (e.g., “Cristiano Ronaldo plays for ___”)
as well as the correct answer for the years be-
tween 2010 and 2020. DATESET, described in
Appendix D, is also generated through a series
of templates, but populated using a combination
of random dates/durations (e.g., “What day of the
week was it 30 days ago?”). Critically, knowing the
current date is required to answer these questions.

Model
GPT-J
GPT-J + CC
Toolformer (disabled)
Toolformer
OPT (66B)
GPT-3 (175B)

TEMPLAMA DATESET

13.7
12.9
12.7
16.3
14.5
15.5

3.9
2.9
5.9
27.3
1.3
0.8

Table 7: Results for the temporal datasets. Toolformer
outperforms all baselines, but does not make use of the
calendar tool for TEMPLAMA.
Toolformer