This is awesome. Thank you.
Is there a way to decouple this
embedding = nn.Embedding(n_embed, embed_dim).from_pretrained(TEXT.vocab.vectors)
from TEXT? This essentially means your embedding layer is a function of your training data. I want to treat my embedding layer as fixed.
Basically, I want to be able to put
embedding = nn.Embedding(n_embed, embed_dim).from_pretrained(..)
in a nn.Module class that loads my pretrained embeddings.
Here’s what I’ve tried:
dummy_text = data.Field()
dummy_text.build_vocab(vectors=vectors)
dummy_text.vocab.set_vectors(stoi=vectors.stoi, vectors=vectors.vectors, dim=vectors.dim)
embedding = nn.Embedding.from_pretrained(torch.FloatTensor(dummy_text.vocab.vectors))

But dummy_text is not being updated. The output of dummy_text.vocab.itos is ['<unk>', '<pad>']. So obviously when I running anything through my embedding layer I’m getting index out-of-bounds errors.
What’s the correct design pattern here? Do I just keep the embedding layer outside my nn.Module class? What would I do at inference time if the embedding layer is outside my nn.Module class? Do I need to use the training data to reconstruct my embedding layer to make predictions on new, unseen data points? Surely, there is a better way.
Thanks for your help.