Sparse gradients mode can be enabled for nn.Embedding, with it gradient elementwise mean & variance estimates are updated correctly (for specific optimizers); but this may not reduce peak memory, as that gradient tensor is short-lived (due to reverse mode backprop). Instead, it is possible to not move nn.Embedding to GPU (moving just a lookup results in forward() instead).