The methods they use in the following paper might help perhaps.

  
      
      arXiv.org
  
  
    

Understanding Black-box Predictions via Influence Functions

How can we explain the predictions of a black-box model? In this paper, we
use influence functions -- a classic technique from robust statistics -- to
trace a model's prediction through the learning algorithm and back to its
training data, thereby...