'lm_head.weight' is improperly not initialized when loading BART weights into BartForCausalLM
### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.6.0
- PyTorch version (GPU?): 1.11.0+cu113 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@patrickvonplaten 
@patil-suraj 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Input: `bart = BartForCausalLM.from_pretrained('facebook/bart-base')`
Output: `Some weights of BartForCausalLM were not initialized from the model checkpoint at facebook/bart-base and are newly initialized: ['lm_head.weight']`

Input: `bart2 = BartForConditionalGeneration.from_pretrained('facebook/bart-base')`
Result: The LM Head for the encoder decoder model is properly initialized from the Bart-Base checkpoint

### Expected behavior


For both model configurations using `facebook/bart-base`, the LM Head layer has the same dimensions (`(lm_head): Linear(in_features=768, out_features=50265, bias=False)`). However, for BartForCausalLM, the LM head is randomly initialized, while for BartForConditionalGeneration, the LM is properly instantiated from the `facebook/bart-base` checkpoint.

Isn't this incorrect?

Hey @nbravulapalli,

Thanks for the report. The model is actually correctly initialized, the reason you see this error message is simply because these lines:
https://github.com/huggingface/transformers/blob/56f50590d5a9ac881db9ee1753f4642cf3d33d28/src/transformers/models/bart/modeling_bart.py#L1275
are preset for `BartForConditionalGeneration` but not for `BartForCausalLM`
https://github.com/huggingface/transformers/blob/56f50590d5a9ac881db9ee1753f4642cf3d33d28/src/transformers/models/bart/modeling_bart.py#L1700

The model should however be correctly initialized. 

Do you mind opening a PR to fix it? :-)
<|||||>Thank you for your reply @patrickvonplaten! I will take a shot at the PR, but I have two clarifying questions:

1) If I understand correctly, the LM head is properly initialized for both BartForConditionalGeneration and BartForCausalLM, but with BartForConditionalGeneration the error message is suppressed?

This is a separate question:
2) When I evaluate `bartMod.config.add_cross_attention` (a BartForCausalLM object) I get `False`. However, the model structure for `bartMod` includes
`(encoder_attn): BartAttention(`
`(k_proj): Linear(in_features=768, out_features=768, bias=True)`
`(v_proj): Linear(in_features=768, out_features=768, bias=True)`
`(q_proj): Linear(in_features=768, out_features=768, bias=True)`
`(out_proj): Linear(in_features=768, out_features=768, bias=True))`
which I assumed was the cross-attention layer designed for the decoder query matrix and the encoder key, value matrices.

a) Is this cross-attention layer not actually present in the CausalLM model, and this layer is improperly displayed? If so, does this mean that the CausalLM model doesn't actually work out of the box (since the built-in cross-attention layers are removed), and requires finetuning to be used for CausalLM?

b) If this cross-attention layer is actually present in the CausalLM model (and this layer is properly displayed), then how is this cross-attention layer still working even without being able to receive the encoder key, value matrices at inference time?<|||||>1. Yes, note it's a warning message that is suppressed not a error message

a) Yes BartForCausalLM won't work out of the box exactly because the cross attention layers are removed
b) It's not present in BartForCausalLM :-)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.