  Convolutional neural networks (CNNs) offer great machine learning performance
over a range of applications, but their operation is hard to interpret, even
for experts. Various explanation algorithms have been proposed to address this
issue, yet limited research effort has been reported concerning their user
evaluation. In this paper, we report on an online between-group user study
designed to evaluate the performance of "saliency maps" - a popular explanation
algorithm for image classification applications of CNNs. Our results indicate
that saliency maps produced by the LRP algorithm helped participants to learn
about some specific image features the system is sensitive to. However, the
maps seem to provide very limited help for participants to anticipate the
network's output for new images. Drawing on our findings, we highlight
implications for design and further research on explainable AI. In particular,
we argue the HCI and AI communities should look beyond instance-level
explanations.

Human-Computer Interaction