Hi,
The problem is that even when I comment out the word2vec embedding, which is a constant since I don’t require gradient for it, and optimize over everything else, I expect this is the same as not commenting out it (because the optimizer has nothing to do with it), it still gave me speed up. So I think even when requires_grad is false, the optimizer, for some reason, still looks at it?