Hidden State Embedding-Transformers
Hi everybody, I want to use Bert model to get the embedding for a sentence after I fine-tuned it with raw texts. I was wondering if is that possible or not and anybody can help me with that? 
Hi! Did you have a look at [this older issue](https://github.com/huggingface/transformers/issues/1950)?<|||||>> Hi! Did you have a look at [this older issue](https://github.com/huggingface/transformers/issues/1950)?

Yes I did, but my concern is that if I want to fine-tune it on my raw text data (language model with LM head) then how should I use it for sentence embedding? Can I just remove the LM head of it?<|||||>I think this is not necessary since, according to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm), you can access every hidden states of your model without removing the LM head.<|||||>> I think this is not necessary since, according to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm), you can access every hidden state of your model without removing the LM head.

Yes, thank you for your reply, I figured it out the language model network can be split into two parts which are 
```
self.bert = BertModel(config)
self.cls = BertOnlyMLMHead(config)
```
then I just need to get the output from self.bert if I want to access the hidden states.