  Artificial Intelligence in Medicine has made significant progress with
emerging applications in medical imaging, patient care, and other areas. While
these applications have proven successful in retrospective studies, very few of
them were applied in practice.The field of Medical AI faces various challenges,
in terms of building user trust, complying with regulations, using data
ethically.Explainable AI (XAI) aims to enable humans understand AI and trust
its results. This paper presents a literature review on the recent developments
of XAI solutions for medical decision support, based on a representative sample
of 198 articles published in recent years. The systematic synthesis of the
relevant articles resulted in several findings. (1) model-agnostic XAI
techniques were mostly employed in these solutions, (2) deep learning models
are utilized more than other types of machine learning models, (3)
explainability was applied to promote trust, but very few works reported the
physicians participation in the loop, (4) visual and interactive user interface
is more useful in understanding the explanation and the recommendation of the
system. More research is needed in collaboration between medical and AI
experts, that could guide the development of suitable frameworks for the
design, implementation, and evaluation of XAI solutions in medicine.

Artificial Intelligence;Machine Learning