T5ForConditionalGeneration lm_head not initialized from pretrained checkpoint
It seems that pretrained T5 checkpoints such as `"t5-small"`, `"t5-base"`, etc., do not contain the `lm_head` parameter.
https://github.com/huggingface/transformers/blob/84ea427f460ffc8d2ddc08a341ccda076c24fc1f/src/transformers/models/t5/modeling_t5.py#L1434-L1438
@patrickvonplaten explained in #3553 that these three parameters are shared, but, while I agree that the encoder and decoder embeddings are shared, I don't think `lm_head` is also shared with them. This is because (1) I couldn't find any such sharing logic in `modeling_t5.py`, and (2) these weights are separate in the original TF checkpoint.
Hey @ZhaofengWu, 

For all encoder-decoder models, such as T5, Bart, ..., the `lm_head` of the decoder is shared between the decoder input word embedding if `tie_word_embeddings` is set to `True` in the model's config. You can check the config for `t5-small` [here](https://huggingface.co/t5-small/blob/main/config.json) *e.g.*
If the config doesn't have this parameter set, then one should look at the default value in `configuration_utils.py` here: https://github.com/huggingface/transformers/blob/c1125dc2ba9f3c383bf860ac9fcd67268385ad8d/src/transformers/configuration_utils.py#L256 where it is set to ``True``.<|||||>In TF the weighs are also shared <|||||>Ahh ok, thanks for pointing me to where the tying takes place. Also I didn't realize that T5 v1.1 and T5 have different behaviors here, and had this question since I saw separate parameters in the TF checkpoint for v1.1.