New config param for cross-attention dimensionality
# 🚀 Feature request

Add a new config param `n_cross` for each model that has cross-attention. This param will determine key and value matrices dimensionality in cross-attention.

## Motivation

I have pretrained encoder (hidden size 512) and want to combine it with GPT-2 medium (hidden size 1024) by cross-attention. Now I cant do it, because in according with this PR https://github.com/huggingface/transformers/commit/1d6e71e1167dea9e026391ec5a1a2d7ec33d22af query and key matrices of cross-attention have dimensionality same as self-attention.

In code it could looks like:
```
config.is_cross_attention = True
config.n_cross = 512
```
And in doc string:
```
n_inner (:obj:`int`, optional, defaults to None):
                Dimensionality of the inner feed-forward layers. :obj:`None` will set it to 4 times n_embd.
n_cross (:obj:`int`, optional, defaults to None):
                Dimensionality of the cross-attention input. :obj:`None` will set it same as n_embd.
```

## Your contribution

Actually I have already fixed this in my fork, but I didn't make any tests and docs. But I would be able to prepare PR within next two weeks.

is this different than d_model/hidden_size?<|||||>Yes it is. As I undestand from T5 and TransformerXL config this param named `n_embd` in GPT2 config (embedding and hidden size dimensionality). It has impact on selt-attention dimentionality:
Q (n_embd x n_embd) 
K (n_embd x n_embd) 
V (n_embd x n_embd). 

For now cross-attention dimensionality same, but I want:
Q (n_embd x n_embd)
K (n_cross x n_embd) 
V (n_cross x n_embd) 
in cross-attention.<|||||>Hey @Squire-tomsk, thanks for your issue. This can easily be resolved by introducing a new config parameter for each model that can be used as a decoder in the `EncoderDecoder` Framework, so probably in `configuration_utils.py`. By default it can be set to `d_model` or `hidden_size`, if not further specifiied.

IMO, adding another config param for this case is OK. What do you think? @sshleifer, @sgugger, @LysandreJik ? <|||||>As long as the parameter is set to the right default and doesn't break backward compatibility, I have no objection.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi,

This feature has recently been implemented in #13874 . We called the attribute `cross_attention_hidden_size`.

In short, there are 2 options:
- either you set it to `None`, in which case the `encoder_hidden_states` will be projected using a single linear layer, to match the hidden size of the decoder (in case they don't match).
- either you set it to the size of the encoder, in which case the decoder will project the `encoder_hidden_states` to the same dimension as the decoder when creating `keys` and `values` in each cross-attention layer. This is the case for the recently added TrOCR model.