But that’s the thing. I set requires_grad to False when using pretrained embeddings, even before I removed the nn.Embedding layer, because I didn’t want the pretrained Wiki2Vec embeddings to update. Now that I have removed that layer, and I include the Wiki2Vec embeddings in the input only, I observe a substantial decrease in performance.
And I can’t figure out why.