`decoder` without bias in BertLMPredictionHead
## ❓ Questions & Help

What does this comment mean?

https://github.com/huggingface/transformers/blob/80bf868a268fa445926bc93f7fe15960853e828e/transformers/modeling_bert.py#L394-L407
Hi, I believe that means that the decoder is a linear layer that has the same weights as the word embedding matrix. However, that embedding matrix does not have a bias, whereas the decoder does have a bias. 

It is initialized to a vector of zeros here, but it can update its weights during training and has actual values in pre-trained models. For example:

```py
from transformers import BertForMaskedLM

bert = BertForMaskedLM.from_pretrained("bert-base-cased")
print(bert.cls.predictions.bias)

# tensor([-0.1788, -0.1758, -0.1752,  ..., -0.3448, -0.3574, -0.3483], requires_grad=True)
```<|||||>Oh, I see. That makes sense. So it should share parameters with the embedding weights? Where is that enforced in the code?<|||||>Yes, exactly. You can see it in the [bert_modeling.py file, inside the BertForMaskedLM class](https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L754-L759).<|||||>Thanks for the clarification!<|||||>hi, 
I know it's an old issue but I had the same questions:
* it's not clear to me in the code where this parameter sharing is enforced
* is there any intuition why it is done ?  

thanks in advance <|||||>@thibault-formal Hey, I had similar questions and asked them in the huggingface forum, I think my [post there](https://discuss.huggingface.co/t/understanding-bertlmpredictionhead/3618) could be helpful for you (if its still relevant). Your first point should be adressed by the explanation of my understanding and the second one is adressed by both replies.
Cheers!