Well, just to see if it work, stay in the training mode with model.train() so you can perform backward(). Just avoid doing the optimizer step to actually change the trainable parameters. I can see no harm here.
Apart from that, I’ve tried to visualize the importance of words when using an RNN using attention, where the attention weights indicate which word had the most affect on the prediction. It’s not unintuitive but has been shown to have problems 14.