• Figure 10 presents an example of L1-
normalized gradient attributions for a single
instance (§4.2).

• Figure 11 presents a comparison of attribution
methods on the ROAR reliability check (§4.2)
for CoS-E v1.0.

• Figure 12 presents results of the ROAR fea-
ture importance agreement measure (§4.2) for
CoS-E v1.11 and E-SNLI.

Dataset

Num. Instances
Train-Val-Test

Input Length

Extractive Rationale

# Tokens

# Tokens

% of doc.

Natural Language Rationale
# Tokens

% of doc.

E-SNLI
CoS-E v1.0
CoS-E v1.11

549,367-9,842-9,824

7,610-950-none
9,741-1,221-none

20.27 +/– 6.95
13.69 +/– 5.97
13.40 +/– 5.77

4.01 +/– 3.01
4.57 +/– 4.16
6.80 +/– 5.79

21.30 +/– 15.82
35.36 +/– 27.22
53.24 +/– 36.05

12.39 +/– 6.43
12.74 +/– 6.99
6.97 +/– 4.14

65.67 +/– 35.46
108.18 +/– 77.26
58.01 +/– 39.60

Table 7: Statistics on datasets with ground-truth rationales. Results are presented as mean (one standard deviation)
on the training set. CoS-E does not contain test set rationale annotations.
Measuring Association Between Labels and Free-Text Rationales