  Radio galaxy morphological classification is one of the critical steps when
producing source catalogues for large-scale radio continuum surveys. While many
recent studies attempted to classify source radio morphology from survey image
data using deep learning algorithms (i.e., Convolutional Neural Networks), they
concentrated on model robustness most time. It is unclear whether a model
similarly makes predictions as radio astronomers did. In this work, we used
Local Interpretable Model-agnostic Explanation (LIME), an state-of-the-art
eXplainable Artificial Intelligence (XAI) technique to explain model prediction
behaviour and thus examine the hypothesis in a proof-of-concept manner. In what
follows, we describe how \textbf{LIME} generally works and early results about
how it helped explain predictions of a radio galaxy classification model using
this technique.

Instrumentation and Methods for Astrophysics;Astrophysics of Galaxies