Hi @matthew,
I’ve looked into this a bit in the past and your intuitions are nearly spot on! What you have described is something like a saliency map. Here are some references that might be useful:

https://arxiv.org/pdf/1312.6034.pdf 23
https://arxiv.org/pdf/1703.01365.pdf 13
https://arxiv.org/pdf/1706.03825.pdf 12

Taking the simplest case, what is effectively done is a forward pass, then a backward pass all the way to the input layer. The intuition is that will give you a sense of which tokens have the largest impact on the output if you were to change them by a small amount. In other words, if one of the input tokens from a sentence about cats is changed in such a way that the output of the model becomes 0 (i.e. a sentence about dogs), then that particular token would have a high saliency score.
Some of the papers above still were a bit unclear to me, so here are some supplementary references to hopefully get you on your way:

https://medium.com/@thelastalias/saliency-maps-for-deep-learning-part-1-vanilla-gradient-1d0665de3284 36
https://glassboxmedicine.com/2019/06/21/cnn-heat-maps-saliency-backpropagation/ 12
https://blog.qure.ai/notes/visualizing_deep_learning 17