Hello Everyone,
I am fine-tuning a pertained masked LM (distil-roberta) on a custom dataset. Post-training, I would like to use the word embeddings in a downstream task. How does one go about obtaining embeddings for whole-words when the model uses sub-word tokenising.  For example, tokeniser.tokenize(‘floral’) will give me [‘fl’, ‘oral’]. So, if ‘floral’ is not even a part of the vocabulary, how do I obtain its embedding?
When I do this:
tokens = tokenizer.encode("floral")
word = tokenizer.encode("floral",return_tensors='pt')
output = model(word)

I see that output is a tensor with shape torch.Size([1, 4, 50265]) and rightly so, because a LM will output the probability distribution across all the words in the vocabulary. I am expecting something like [1,768]. Can someone please help?