Solved by nielsr in post #2 
              
              
                Hi, 
Yes that’s possible. There are some cool libraries out there that can be used to do that: 

ELI5: GitHub - TeamHG-Memex/eli5: A library for debugging/inspecting machine learning classifiers and explaining their predictions

LIME: GitHub - marcotcr/lime: Lime: Explaining the predictions of any m…