Both embeddings take only 0.008 ms

This is with synchronization?
If you have big embedding, the difference might also be that the gradients are dense by default. See the “sparse” arg for the Embedding layer.