  Counterfactual explanations are increasingly used as an Explainable
Artificial Intelligence (XAI) technique to provide stakeholders of complex
machine learning algorithms with explanations for data-driven decisions. The
popularity of counterfactual explanations resulted in a boom in the algorithms
generating them. However, not every algorithm creates uniform explanations for
the same instance. Even though in some contexts multiple possible explanations
are beneficial, there are circumstances where diversity amongst counterfactual
explanations results in a potential disagreement problem among stakeholders.
Ethical issues arise when for example, malicious agents use this diversity to
fairwash an unfair machine learning model by hiding sensitive features. As
legislators worldwide tend to start including the right to explanations for
data-driven, high-stakes decisions in their policies, these ethical issues
should be understood and addressed. Our literature review on the disagreement
problem in XAI reveals that this problem has never been empirically assessed
for counterfactual explanations. Therefore, in this work, we conduct a
large-scale empirical analysis, on 40 datasets, using 12 explanation-generating
methods, for two black-box models, yielding over 192.0000 explanations. Our
study finds alarmingly high disagreement levels between the methods tested. A
malicious user is able to both exclude and include desired features when
multiple counterfactual explanations are available. This disagreement seems to
be driven mainly by the dataset characteristics and the type of counterfactual
algorithm. XAI centers on the transparency of algorithmic decision-making, but
our analysis advocates for transparency about this self-proclaimed transparency

Artificial Intelligence