T5 Models All of the models in this work are
based on T5, though our methods can in principle
be applied to any architecture. The base version
of T5 is a 220M-parameter transformer encoder-
decoder (Vaswani et al., 2017). To carry out su-
pervised finetuning, T5 is trained by maximizing
the conditional likelihood of the correct text output
(from annotated data), given the text input.

We finetune five T5-Base models for each
dataset, supervising with ground-truth labels and
rationales (further details in Appendix A.3-A.4):
• I→R, which maps task inputs to rationales,
without ever being exposed to task outputs.
• R→O, which maps rationales to task outputs.
The only input elements this model is exposed
to are answer choices (for CoS-E).
• I→OR, which maps inputs to outputs and ra-
tionales.
3CoS-E does not contain test set rationale annotations, so

we report performance values on the validation set.

CQA

SNLI

SST AgNews Evidence Movie MultiRC LGD 20 News Amazon

Beer

BoolQ FEVER
Measuring Association Between Labels and Free-Text Rationales