bert2gtp_neo
### System Info

- `transformers` version: 4.24.0.dev0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

I'm trying to build an encoder-decoder model using `bert as an encoder` and `gpt_neo as a decoder`. I finetune it on the `summarization` task.
I got this error at the beginning of the training 
`TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'`
However, when I used the gpt2 it worked well. I'm sharing the [Colab notebook for more details](https://colab.research.google.com/drive/16pg7XRDJ6iu4ih15qZAx3E09cnk_j77u?usp=sharing)
@patil-suraj, @sgugger, @patrickvonplatenit

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

 [Colab notebook for more details](https://colab.research.google.com/drive/16pg7XRDJ6iu4ih15qZAx3E09cnk_j77u?usp=sharing)

### Expected behavior

Helping to resolve the problem
@patil-suraj 
@sgugger 
@patrickvonplaten <|||||>Hey @elmadany,

In general, could you maybe share a short reproducible code snippet below that is easy to copy-paste? 
The google colab is more than just a error production and going through it step-by-step sadly takes us too much time. Thank you :-)

To answer your question: GPT-Neo is currently not compatible to be used as a decoder in an encoder-decoder setting. GPT-Neo  does not have a `encoder_hidden_states` function argument here: https://github.com/huggingface/transformers/blob/371337a95b5d82cc9376c2595ed2022a5eb2ee6e/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L721

whereas GPT-2 has one: https://github.com/huggingface/transformers/blob/371337a95b5d82cc9376c2595ed2022a5eb2ee6e/src/transformers/models/gpt2/modeling_gpt2.py#L1030

=> that's the reason why GPT-2 works but GPT-Neo doesn't. 

To solve this one would need to adapt GPT-Neo to allow for cross-attention, similar to: https://github.com/huggingface/transformers/pull/6415 . Feel free to give it a try if you would like to add a feature to transformers :-) 
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>> Hey @elmadany,
> 
> In general, could you maybe share a short reproducible code snippet below that is easy to copy-paste? The google colab is more than just a error production and going through it step-by-step sadly takes us too much time. Thank you :-)
> 
> To answer your question: GPT-Neo is currently not compatible to be used as a decoder in an encoder-decoder setting. GPT-Neo does not have a `encoder_hidden_states` function argument here:
> 
> https://github.com/huggingface/transformers/blob/371337a95b5d82cc9376c2595ed2022a5eb2ee6e/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L721
> 
> whereas GPT-2 has one:
> 
> https://github.com/huggingface/transformers/blob/371337a95b5d82cc9376c2595ed2022a5eb2ee6e/src/transformers/models/gpt2/modeling_gpt2.py#L1030
> 
> => that's the reason why GPT-2 works but GPT-Neo doesn't.
> 
> To solve this one would need to adapt GPT-Neo to allow for cross-attention, similar to: #6415 . Feel free to give it a try if you would like to add a feature to transformers :-)

Hello, have u ever tried the `generate() ` method using  `encoder_hidden_states ` parameter, like:

```
model = GPT2LMHeadModel.from_pretrained('gpt2', output_hidden_states=True, add_cross_attention=True)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

memory = ["Oh no", "Oh no", "Excellent !"]
memory = [tokenizer.encode(i, max_length=3) for i in memory]
memory = torch.tensor(memory)
memory = model(memory, past_key_values=None)
memory = memory.hidden_states[0]      # batch_size, 1, 768
 
inputs = ["This is my", "I want to", "The world is"]
inputs = [tokenizer.encode(i, max_length=3) for i in inputs]
inputs = torch.tensor(inputs)


output = model.generate(encoder_hidden_states=memory, inputs=inputs)
print(tokenizer.batch_decode(output))

output = model.generate(inputs=inputs)
print(tokenizer.batch_decode(output))
```

This is just a demo, I found that the outputs are the same, no matter how I set the parameter `encoder_hidden_states` is set, and in the debug mode, I found that this parameter is not passed to `GPT2LMHeadModel`