rationalizing models, which are fully differentiable
and jointly predict the task output with the ratio-
nale; and pipelines, which rationalize first and then
predict task output with a separate model. We first
show that, for CommonsenseQA and SNLI, a self-
rationalizing model provides rationales that better
indicate the correct label than a pipeline (§3.1).
Next, we show that sufficiency is not universally
applicable: a natural language rationale on its own
does not generally provide enough information to
arrive at the correct answer (§3.2). These findings
suggest that a faithful-by-construction pipeline is
not an ideal approach for reasoning tasks, leading
us to ask: is there is a way to achieve faithful free-
text rationalization with self-rationalizing models?
Measuring Association Between Labels and Free-Text Rationales