Figure 3: An example of a joint architecture (I→OR; §2) for CoS-E v1.0 with a human-written rationale. Trained
on both task signal and human rationales, these models are effective at generating fluent rationales while retaining
good task performance (Table 9-10 in Appendix B).

R→O with R∗
97.67
84.84
68.63

IR→O with R∗
98.72
90.42
80.84

Model
∆
E-SNLI
+1.05
CoS-E v1.0
+5.58
CoS-E v1.11
+12.21
Table 5: A comparison of the IR→O and R→O mod-
els (§2) evaluated with ground-truth natural language
rationales (R∗). In some cases accuracy improves sub-
stantially with the addition of the input, indicating that
rationales are not always sufficient and pipelines are not
always effective.

nale quality metric (Table 4). The fact that the
pipeline’s strong performance does not generalize
to a complex prediction task such as CoS-E empiri-
cally demonstrates that training on label signal O is
important to generate good-quality rationales and
avoid cascading errors.
Measuring Association Between Labels and Free-Text Rationales