I’m not exactly sure how to do this but I think what you are basically trying to do is to find how strongly the decoder’s attention mechanism “attends” to each token that comes in from the encoder. An example is presented here in this blog post 1;

image1228×601 89.8 KB

Based on this idea, you could try taking the final transformer stack in the decoder and visualize the attention weights where it attends to the encoder inputs.
I just googled and found this library that does this for you - link 9.