Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling
Ji, Xiapu Luo, and Ting Wang. 2020. Interpretable
deep learning under fire. In 29th USENIX Security
Symposium (USENIX Security).

Haizhong Zheng, Earlence Fernandes, and Atul Prakash.
2019. Analyzing the interpretability robustness of
self-explaining models. In ICML 2019 Security and
Privacy of Machine Learning Workshop.

A Additional Information
A.1 Overview of Prior Work on Pipelines
In Table 2, we overview the datasets and types of
rationales used in prior work on pipelines. The
sources of datasets are: CommonsenseQA (Tal-
mor et al., 2019), SNLI (Bowman et al., 2015),
SST (Socher et al., 2013), AgNews (Del Corso
et al., 2005), Evidence Inference (Lehman et al.,
2019), Movie Reviews (Zaidan and Eisner, 2008),
MultiRC (Khashabi et al., 2018), LGD (Linzen
et al., 2016), 20 News (Lang, 1995), Amazon Re-
views (McAuley and Leskovec, 2013), Beer Re-
views (McAuley et al., 2012), BoolQ (Clark et al.,
2019), FEVER (Thorne et al., 2018).
Measuring Association Between Labels and Free-Text Rationales