A.2 Details of Datasets
We summarize dataset statistics in Table 7. The
two versions (v1.0, v1.11) of CoS-E correspond
to the first and second versions of the Common-
senseQA dataset. CoS-E v1.11 has some noise in
its annotations (Narang et al., 2020).10 This is our
primary motivation for reporting on v1.0 as well,
which we observe does not have these issues.

A.3 Details of T5
The T5 model (Raffel et al., 2020) is pretrained on a
multi-task mixture of unsupervised and supervised
tasks, including machine translation, question an-
swering, abstractive summarization, and text classi-
fication. Its inputs and outputs to every task are text
sequences; we provide the input-output formatting
for training and decoding of our T5 models in Ta-
ble 8. T5 can provide any word in the vocabulary
as an answer.

Implementation Details
Measuring Association Between Labels and Free-Text Rationales