A.5 Note on Robustness Equivalence

Convergence

Worst-case model performance under large noise
values in the Robustness Equivalence experiments
(Figures 4 and 8) reaches 0 rather than random
accuracy due to structure of the models’ output.
The I→OR model is trained to produce a delimiter
to distinguish the label from the rationale in a long
string of output tokens. When it fails to produce
the delimiter under high noise, we cannot delineate
the label from the rationale in the output where

10https://github.com/salesforce/cos-e/

11https://huggingface.co/docs/datasets/

issues/2

master/#

• Tables 11-13 contain additional validation set
examples (non-cherry-picked) of noised in-
puts for CoS-E v1.0 (§4.1).

multiple answer choices are often mentioned, so
we mark the label as incorrect.

A.6 Further Discussion of Rationale Quality

Metric
Measuring Association Between Labels and Free-Text Rationales