Unable to pass encoder_outputs to generate calls
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: Github Main branch

### Who can help
 TextGeneration: @TevenLeScao 
 

## Information

Model I am using (Bert, XLNet ...): T5

I'm unable to pass preconputed encoder_outputs to the `.generate()` method.

I've tried defining:

```python
model_kwargs = {"encoder_outputs": encoder_outputs}
output = model.generate(model_kwargs)
```
But I've noticed some validation errors for `input_ids`. Even if I replace input_ids with dummy values, I've noticed the model_kwargs is always replaced here:

https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L448

I think it can be fixed (without optimization) by just replacing:

```python
model_kwargs["encoder_outputs"] = encoder_outputs
```

with:

```python
model_kwargs.setdefault("encoder_outputs", encoder_outputs)
```

If you agree I can try to open a PR to fix this.

Best,
Hey @gabisurita - I understand that one might want to forward encoder_outputs in the generate function. However, adding such a possibility opens the door for many problems in case `beam_search` is chosen. We are currently working on a bigger refactor that should solve this problem by a better design choice of `generate()`. I'm afraid that this will still take ~3,4 weeks to complete though.<|||||>Hi @patrickvonplaten,

I've noticed that your PR refactoring generate was merged. It seems a big improvement, thank you! Still, `input_ids` is still required or overridden by an empty tensor. It's still not clear to me how can I use the new API with `encoder_outputs` or `imput_embeds`.<|||||>Hey @gabisurita - if you look at the tests, you can now directly use `beam_search` instead of generate for your use case I think :-). Here the tests: https://github.com/huggingface/transformers/blob/a7d73cfdd497d7bf6c9336452decacf540c46e20/tests/test_generation_utils.py#L295 

From the tests, it should be quite easy to understand how to use `beam_search` directly I think :-) 

Let me know if that helps!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.