Hi,
Yes that’s possible. There are some cool libraries out there that can be used to do that:

ELI5: GitHub - TeamHG-Memex/eli5: A library for debugging/inspecting machine learning classifiers and explaining their predictions

LIME: GitHub - marcotcr/lime: Lime: Explaining the predictions of any machine learning classifier


I believe it requires some tweaks to make it work on Transformer-based models, but I did this myself some time last year, so it’s definitely possible.
Edit: this Github thread 3 might be helpful.