ues of k for which both attributions have passed
the inflection point, the “random” line should be
below the attribution line, assuming that after this
point, a noisier rationale is more similar is IR→O
to I→O and hence the score is closer to 0. Both
criteria hold for attributions from the rationale log-
its, a(X)R (green vs. blue line in Figure 7, right)
for CoS-E v1.0 and other datasets (see Figure 12
in Appendix B). This reliability check confirms
that gradient-attribution works well in our setting.
Agreement Method and Results To measure
feature importance agreement—whether tokens im-
portant for label prediction are important for ra-
tionale generation (and vice versa)—we repeat the
same experiment, but measure performance with re-
spect to the other output’s metric. For attributions
computed from label logits, a(X)L, we measure
the effect of their occlusion on rationale quality us-
ing the rationale quality score. For attributions with
Measuring Association Between Labels and Free-Text Rationales