Can I save a word embedding from BERT and used again later for computational purpose(as BERT takes much more time). Just like in Glove. If yes then How? and is this good idea to do so?
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

As per my understanding, you can! 

If you read the BERT paper by Devlin et.al., you can see that two suggested ways to extract said word embeddings would be to concatenate the last four hidden layers (9 to 12), generating a 4*768=3072 sized embedding for each token. Alternatively you can also sum or average out the last 4 layers to generate vectors of size 768.

You can also instead prefer storing sentence embeddings, then the CLS token serves as averaged representation of the sentence embedding (not a very good representation though, unless the model was fine-tuned well on the data or the LM was pretrained on similar data, and even then the sentence embeddings through that CLS token could be substandard). Although, averaging out the second to last hidden layer (i.e. averaging out token embeddings for the sentence at the 11th hidden layer) seems to generate decent sentence embeddings (of size 768, the same as one token, since its averaged and not concatenated). 

Hope this helps!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
