Hi,
I assume that the prediction of your model is differentiable with respect to the input. If so, you can compute the expectation of the absolute value of the gradient for the variables. Check this paper out: https://ieeexplore.ieee.org/document/8999179 61.