By examining the regions of largest slope, we
gain insights into model behavior. On the ratio-
nale quality measure, both versions of CoS-E’s
rationales reach a minimum contribution to task ac-
curacy at σ2 = 20 (Figure 5). We similarly observe
the largest drop in task accuracy (Figure 4) for CoS-
E v1.0 between σ2 = 15 to σ2 = 20.9 Thus, at
lower noise levels (0–15), the model exhibits both
stable labels and rationales, and at higher levels
(20+), both unstable, indicating robustness equiv-
alence. Similar conclusions can be reached for
E-SNLI and CoS-E v1.11; we conclude that the
I→OR model demonstrates high label-rationale as-
9See Appendix A.5 for a note on why the I→OR models

achieve worse-than-random accuracy at high values of σ2.

Figure 4: Results of the label portion of the robustness
equivalence test for CoS-E v1.0.

Figure 5: Results of the rationale portion of the robust-
ness equivalence test for both CoS-E datasets.
Measuring Association Between Labels and Free-Text Rationales