I would probably use the torch.autograd.grad 7 function.
Say you have the inputs to the LSTM as features (if you don’t train anymore and want to have grads, you would need to re-wrap it as features_needing_grad = Variable(features.data, requires_grad=True) or so) and the score of the predicted class as score[i], you should be able to do
grads = torch.autograd.grad(score[i], features_needing_grad)

and use these gradients.
Best regards
Thomas