Hi,
Could you send me a full script that runs with all the sizes being the ones you use. Replacing your data with random tensors.
I guess two possible things here:

The graph expands accross iterations and thus the traversal for the backward becomes dead slow. I need the full running code to check that.
Give the simplicity of your graph and the fact that it’s mainly Embedding from what I see, you might be hitting a very bad worst case. Here again, try sparse=true for the embedding, it was made for that purpose.