RORA: Robust Free-Text Rationale Evaluation
<|reference_start|>RORA: Robust Free-Text Rationale Evaluation: Free-text rationales play a pivotal role in explainable NLP, bridging the knowledge and reasoning gaps behind a model's decision-making. However, due to the diversity of potential reasoning paths and a corresponding lack of definitive ground truth, their evaluation remains a challenge. Existing evaluation metrics rely on the degree to which a rationale supports a target label, but we find these fall short in evaluating rationales that inadvertently leak the labels. To address this problem, we propose RORA, a Robust free-text Rationale evaluation against label leakage. RORA quantifies the new information supplied by a rationale to justify the label. This is achieved by assessing the conditional V-information \citep{hewitt-etal-2021-conditional} with a predictive family robust against leaky features that can be exploited by a small model. RORA consistently outperforms existing approaches in evaluating human-written, synthetic, or model-generated rationales, particularly demonstrating robustness against label leakage. We also show that RORA aligns well with human judgment, providing a more reliable and accurate measurement across diverse free-text rationales.<|reference_end|>
@article{jiang2024rora:,
    title={RORA: Robust Free-Text Rationale Evaluation},
    author={Zhengping Jiang, Yining Lu, Hanjie Chen, Daniel Khashabi, Benjamin Van
  Durme, Anqi Liu},
    journal={arXiv preprint arXiv:2402.18678},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.18678},
    primaryClass={cs.CL}
}