It seems you are currently reassigning embeddings to the output of the following operations.
Could you use a unique name for the embeddings, which use retain_grad() and return it with the other outputs?
After the backward call, the embeddings.retain_grad() should contain the gradients.