Hm, looks rather alright to me, and your last three print statements are as expected. Maybe just some ideas:


self.word_embeddings = nn.Embedding(...).cuda() should not be needed since from_pretrained creates an Embedding layer and not just sets the weights, at least according to the docs. Of course, it really shouldn’t make a difference. But if all fails, comment it out for test :). It’s a bit strange though, self.embeddings shouldn’t be moved to the GPU after from_pretrained().
*You could give my alternative a chance just to see if there’e any difference
self.word_embeddings.weight.data.copy_(torch.from_numpy(embed_mat))
self.word_embeddings.weight.requires_grad=False/True



What does print(model.word_embeddings.weight[105]) show after calling from_pretrained(). Does it match with emb_mat[105]? Again, it obviously should, but not harm trying.


Maybe make your example minimal with batch_size=1 and seq_len=1, i.e., just [[105]]. Maybe somewhere some shapes are not as expected.


Sorry for not being more helpful!