5.1. Quantitative analysis of short-listed articles

Of the 40 short-listed articles, 24 leveraged KGs in the pre-model phase, nine used KGs in the in-model phase, and eight
used KGs in the post-model phase of XAI system development. It should be noted that one article used KG in both the
pre-model and in-model phases. Most of the machine learning models used in the analysed studies were based on neural
network models. As shown in Figure 2, the largest proportion of studies (23%) used convolutional neural network mod-
els in their XAI system, followed by natural language processing (20%) and deep neural networks (14.3%). In terms of
KG utilisation, 16 studies included the construction of a KG for explainability, while 30 articles explained the utilised
model by feature or relation extraction, and 10 used KGs for inference and reasoning. It should be noted that some arti-
cles used KGs for two or more applications.
Knowledge-graph-based explainable AI- A systematic review