How to average sub-words embeddings to obtain word embeddings?
Hi~

How to average sub-words embeddings to obtain word embeddings?
I only want word-level embedding instead of sub-word-level, how can I get them?

Is there any tokenizer that provides a method that can output the index/mask of sub-words or something?
You may use the word as the input and make the sentence embedding as the word embedding.
for example, input is 
"puppeteer"
tokens as
'[CLS]', 'puppet', '##eer', '[SEP]'
and then get embedding of this tokens list output.<|||||>I have similar usage as well, I did a simple experiment, and observe that the subword embedding [subword1, subword2, subword3...] when input a whole sentence,
 the cosine similarity of [subword1,subword2],[subword1,subword3]... tends to above 90%.
 So that sum and average subwords' embedding doesn't change much.
Btw, I tested this with Roberta models, and I observe quite different result for Bert models.<|||||>Take a look at how bert-sense does it :)
https://github.com/uhh-lt/bert-sense/blob/bfecb3c0e677d36ccfab4e2131ef9183995efaef/BERT_Model.py#L342<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Excuse me did someone solve it ?<|||||>@mathshangw I turn to use the `TransformerWordEmbeddings` of `flair` library to handle this. Here is an [example](https://github.com/flairNLP/flair/blob/master/examples/ner/run_ner.py#L119).<|||||>I found a way to obtain the subtoken mask.

There is an argument called `return_offsets_mapping`. When you pass the tokenized sequence to the tokenizer, the returned offsets mapping records the start position of each token instead of the entire sentence, for example.

```python
tokens: list[str] = 'this is a niceing work'.split()
# NOT THIS => tokens: str = 'this is a niceing work'

tokenizer.tokenize(tokens, add_special_tokens=True)
# ['▁this', '▁is', '▁a', '▁nice', 'ing', '▁work', '</s>', 'en_XX']

tokens = tokenizer(tokens, add_special_tokens=True, is_split_into_words=True,
            return_offsets_mapping=True, return_tensors='pt')
# {
#    'input_ids': tensor([[903, 83, 10, 26267, 214, 4488, 2, 250004]]),
#    'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]]),
#    'offset_mapping': tensor([[[0, 4],
#                               [0, 2],
#                               [0, 1],
#                               [0, 4],
#                               [4, 7],
#                               [0, 4],
#                               [0, 0],
#                               [0, 0]]])}

subtoken_mask = tokens['offset_mapping'][..., 0] != 0
# tensor([[False, False, False, False,  True, False, False, False]])
```

Forget about the weird word *niceing*, I just want to get some sub-tokens. Now, by simply checking if this token starts from the beginning of the given word, we can tell if it is a sub-token.

Hope this is helpful to you guys.