Issue with fp16_opt_level default
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-1091-oem-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, 1 GPU
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 tensorflow: @jplu 
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): BART @sshleifer

The problem arises when using:
* [ ] the official example scripts: 
* [x] my own modified scripts: Running a modified version of finetune.py (via finetune.sh) that specifies "bad word tokens" that BART is prohibited from generating

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: 
* [x] my own task or dataset: Finetuning on a question understanding dataset

## To reproduce

Steps to reproduce the behavior:

1. Run finetune.sh using a GPU with fp16 flag and default settings, instructing the model to perform testing. See set-up below:
Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic

2. At the start of testing, an error related to amp will be encountered:

```python
Traceback (most recent call last):
  File "finetune_prohibition.py", line 444, in <module>
    main(args)
  File "finetune_prohibition.py", line 433, in main
    trainer.test()
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1281, in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in __test_using_best_weights
    results = self.fit(model)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1003, in fit
    results = self.single_gpu_train(model)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 182, in single_gpu_train
    model, optimizers = model.configure_apex(amp, model, self.optimizers, self.amp_level)
  File "/home/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py", line 1006, in configure_apex
    model, optimizers = amp.initialize(model, optimizers, opt_level=amp_level)
  File "/home/lib/python3.6/site-packages/apex/amp/frontend.py", line 358, in initialize
    return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
  File "/home/lib/python3.6/site-packages/apex/amp/_initialize.py", line 171, in _initialize
    check_params_fp32(models)
  File "/home/lib/python3.6/site-packages/apex/amp/_initialize.py", line 87, in check_params_fp32
    name, param.type()))
  File "/home/lib/python3.6/site-packages/apex/amp/_amp_state.py", line 32, in warn_or_err
    raise RuntimeError(msg)
RuntimeError: Found param model.model.shared.weight with type torch.cuda.HalfTensor, expected torch.cuda.FloatTensor.
When using amp.initialize, you do not need to call .half() on your model
before passing it, no matter what optimization level you choose.
```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
When using the new default of fp16_opt_level=O2, this error is encountered - using fp16_opt_level=O1 solves the issue. I'm unsure if this problem is just related to my machine, and would be interested to see whether others can recreate the problem. Although I used modified scripts, the modifications are simple and shouldn't interact with torch.cuda.HalfTensor classes differently. Let me know if you'd like me to try to recreate this issue on an official GLUE/SQUaD task or if you need any other information from me.
I have this issue too, reported to pl https://github.com/PyTorchLightning/pytorch-lightning/issues/2673 
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
