Thanks @ptrblck I’m going to try this today.
Seeing I need to compute gradients for the nn.Embedding on CPU, I’m worried about the runtime.
I’m guessing I should be using SparseAdam with the CPU embeddings
matrix? Would this provide a speedup versus regular Adam? I think theoretically SparseAdam would not need to compute the gradient on the full nn.Embedding matrix, so it should be size_of_matrix/batch_size faster than Adam right (minus some overhead of sparse computation)?