the key to getting speed in these cases is to use sparse gradients. for example nn.Embedding(..., sparse=True). Otherwise, a full dense gradient is computed in each backward, that slows it down significantly.