BART : Cannot run trainer.evaluate() after trainer.train()
## Environment info

- `transformers` version: 4.4.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help

trainer : @sgugger 
bart : @patrickvonplaten 


## Information

Model I am using : bart, barthez, mbart  
I am working on text summarization.

The problem arises when using my own modified script, inspired by the official Seq2Seq example. I am using the Seq2SeqTrainer class.  
I am unable to run trainer.evaluate(...) after trainer.train(...) as well as evaluating the model during training after x epochs or steps.  


## To reproduce

I you want to try it out, here is a link to [my notebook](https://colab.research.google.com/drive/1CqxxM0nOdJRpre_SwLOajl_s9hlKTT9e?usp=sharing)

Steps to reproduce the behavior:

1. Download model, tokenizer, and dataset from hub
2. Run trainer.evaluate(...) (works)
3. Run trainer.train(...) (runs fine)
4. Run trainer.evaluate(...) (returns the error below)

```
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, head_mask, decoder_head_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   1160         elif return_dict and not isinstance(encoder_outputs, BaseModelOutput):
   1161             encoder_outputs = BaseModelOutput(
-> 1162                 last_hidden_state=encoder_outputs[0],
   1163                 hidden_states=encoder_outputs[1] if len(encoder_outputs) > 1 else None,
   1164                 attentions=encoder_outputs[2] if len(encoder_outputs) > 2 else None,

KeyError: 0
```




This is not an issue with Transformers but with using Apex with the "O3" opt-level. This changes your model during the training and results in the error you're seeing. The best is to re-instantiate a clean `Trainer` for evaluation after you're done with training.<|||||>Thank you for your reply, I just tried it and it actually only works with Apex "01" opt-level !