I’ve always been having a hard time reading Pytorch source code, sad.
Could you please tell me the underlying difference between Embedding being sparse or not?
My guess is that, with sparse=True, the forward/backward will only collect the rows from the whole huge Embedding matrix and compute grad for these ones; whereas with sparse=False, forward/backward will do the one-hot multiplication on the whole huge Embedding matrix, which involves much more computation than sparse=True version.