Can we use pre-trained word embeddings for weight initialization in nn.Embedding?
I want to use pre-trained word embeddings as initial weight vectors for embedding layer in a encoder model? How can I achieve this? For example, in Keras, we can pass a weight matrix as parameter to the embedding layer. Is there any similar way in PyTorch to do this?
wasiahmad
you can just assign the weight to the embedding layer. Like: <SCODE>embed = nn.Embedding(num_embeddings, embedding_dim)
# pretrained_weight is a numpy matrix of shape (num_embeddings, embedding_dim)
embed.weight.data.copy_(torch.from_numpy(pretrained_weight))<ECODE>
ruotianluo
I usually use the following way, which is better? <SCODE>#embeddings is a torch tensor.
embedding = nn.Embedding(embeddings.size(0), embeddings.size(1))
embedding.weight = nn.Parameter(embeddings)
<ECODE>
ShawnGuo
I’d say they’re both ok.
apaszke
And how can we keep the embedding matrix fixed during training? I didn’t find that in the doc.
xinyadu
<SCODE>embed.weight.requires_grad = False
<ECODE> to freeze the parameter of it.
cyyyyc123
But when I initialize optimizer, i got “ValueError: optimizing a parameter that doesn’t require gradients”
xinyadu
xwgeng
How can we specifically use glove vectors and mainly in the encoder - decoder model ? Not able to understand.
Navneet_M_Kumar
tushargupta14
What if I want to use sentence embedding as a whole and not word vectors. Suppose I have the sentence embeddings ready, do I create a (number of sentences X sentence embedding dimension) matrix and map it to each sentence id? And pass this matrix to the embedding layer and call the sentence ids in the forward function? Is this approach right? I’m trying to perform a type of sentence classification.
sritvik
Regarding the use of pre-trained word embeddings, the indexes of words that are passed to the Embedding layer should be equivalent to their index in the pre-trained embedding (the numpy matrix), right?
mfa
shirdu
Yes, it’s a matrix. Each row is the embedding for a word. You’ll also want a dictionary: mapping of words (strings) to integers 0, 1, …, N
colesbury
shirdu
shirdu
Yes, the indexes of the words in your vocabulary should be same as your indexes in your embedding numpy matrix. This helps the embedding layer to map the word tokens to vectors.
tushargupta14
austin
Sorry for the bump - what might be the easiest way to have only parts of the embedding matrix frozen? For example, if I wanted to use pre-trained embeddings but for certain words assign a special custom token whose embedding I want to train?
sidedishes
Two ways I’d see doing it: First to have two separate embeddings.  One embedding learns, the other uses pre-trained weights.  Select the embedding to use depending on the value of the input. The other approach would be to overwrite the pretrained parts of the embedding at the beginning of each batch to undo the results of the previous optimizer step.
jlquinn