4The predicted label is from the same system that pro-

duced the predicted rationale.

5Given the large scale of our analysis (>250K instances
evaluated), an automated metric provides coverage, repro-
ducibility and consistency not achievable with human anno-
tation. An author of this paper annotated 60 instances from
both E-SNLI and CoS-E v1.11, and found 82.5% agreement
between their rationale quality score and the automated metric.

Source of Rationales R∗
E-SNLI
CoS-E v1.0
CoS-E v1.11

97.67
84.84
68.63

I→R I→OR
90.52
89.11
62.00
53.47
45.45
53.15

Table 3: Accuracy of the trained R→O model evalu-
ated on ground-truth natural language rationales (R∗)
and rationales generated from two model architectures:
I→OR and I→R (see §2 for model descriptions).

Source of Rationales R∗
7.77
E-SNLI
CoS-E v1.0
21.26
19.09
CoS-E v1.11

I→R
-1.63
-12.11
-12.77

I→OR
-0.86
-6.21
-6.06
Measuring Association Between Labels and Free-Text Rationales