Bert cross attention
## ❓ Questions & Help

In the standard Transformer/Bert architecture, what is the intuition behind cross attention doing a weighted average over the encoder_hidden_states? What happens if we set the value layer to decoder hidden_state instead?

See the cross attention value layer being set to the encoder_hidden_states at [modeling_bert.py#L238](https://github.com/huggingface/transformers/blob/8c67b529f615cc24c46864b8323d2d47a15ccd58/src/transformers/modeling_bert.py#L238), and the weighted average being taken at [modeling_bert.py#L266](https://github.com/huggingface/transformers/blob/8c67b529f615cc24c46864b8323d2d47a15ccd58/src/transformers/modeling_bert.py#L266)

I would imagine the idea is to incorporate a strong presence of the encoder hidden states - else the conditioning on the encoder might be weak. 

We do an attention without the encoder hidden states anyways before the cross attention.