ulatability have moderate to high correlation with
human scores in both an expert and a crowdsourced
setting. In our experiments, model predictions are
often unable to be simulated because they degener-
ate under high values of noise (§4, A.5). We thus
use a variant of this metric that relies on predicting
the gold labels as our measure of rationale quality:
IR→O minus I→O.5 We discuss the effects of this
difference in Appendix A.6.
Measuring Association Between Labels and Free-Text Rationales