Quick question about efficiently initializing model parameters
Hi,

First of all, thanks for your great work on building such an amazing pytorch package!
I am trying to modify the [modeling_bart.py](https://github.com/huggingface/transformers/blob/v4.11.3/src/transformers/models/bart/modeling_bart.py) file, specifically [BartDecoderLayer](https://github.com/huggingface/transformers/blob/v4.11.3/src/transformers/models/bart/modeling_bart.py#L331). I am planning to add a new cross-attention layer which would attend to the graph nodes.  The following snippet is adapted from the original codes. 
```
        self.graph_attn = BartAttention(
            self.embed_dim,
            config.decoder_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=True,
        )
```

My question is about how to **efficiently** initialize these parameters to be the same as the parameters of original BartDecoder Cross-attention layer. By "efficiently", I mean some simple and fast way for the initialization process. A brutal-force way I could easily think of right now is to download [pytorch_model.bin](https://huggingface.co/facebook/bart-large/blob/main/pytorch_model.bin) and then manually assign values by doing key-matching. 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.