  The Rashomon Effect describes the following phenomenon: for a given dataset
there may exist many models with equally good performance but with different
solution strategies. The Rashomon Effect has implications for Explainable
Machine Learning, especially for the comparability of explanations. We provide
a unified view on three different comparison scenarios and conduct a
quantitative evaluation across different datasets, models, attribution methods,
and metrics. We find that hyperparameter-tuning plays a role and that metric
selection matters. Our results provide empirical support for previously
anecdotal evidence and exhibit challenges for both scientists and
practitioners.

Machine Learning;Artificial Intelligence