❓ [BART] Why using bias for LM head if not trained ? 
# ❓ Questions & Help

As I understood, BART is not using a regular Linear layer as LM head, but instead reuse the weights of the shared embeddings.

As show here, biases are added :
https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/src/transformers/modeling_bart.py#L876

But these biases are registered as buffer, not as parameter. **Since they are not trained, they will always stay 0 ?**

If they stay 0, what's the point of having bias at all ?

@sshleifer 
Great Q! The point of using them is that `MarianMTModel`, which inherits from `BartForConditionalGeneration` uses them. You're correct for the bart checkpoints they stay 0. If you think there is a comment or different approach that would be clearer, I'm very open to PR/other ideas.