The computation graph doesn’t seem to be detached, but instead the operations seems to be decrease the gradient magnitude for the attention layers.
The embedding layer (which is a linear layer in fact) also skips the attention module, thus gets larger gradients.
Could you explain your use case a bit as well as the input shapes?
I assume you are feeding a 3-dimensional tensor, since the InstanceNorm layer would raise an error for a 2-dim tensor?