RuntimeError: found torch.cuda.HalfTensor expected torch.cuda.FloatTensor while fine-tuning RAGSequence-base with custom data
## Environment info
     
- `transformers` version: 4.0.0-rc-1 (master)
- Platform: Linux-3.10.0-862.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.5
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help
 RAG: @patrickvonplaten, @lhoestq

## Information
I am fine-tuning RAGSequence-base according to the fine-tuning examples, along with a custom knowledge dataset (<10 lines)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Generated the faiss index and the custom embeddings.
2. Executed `finetune_rag.py` with the same parameters as the given shell script. (except reducing epoch to 1 for test)
3. After going through the specified epoch, it abruptly ends with the following runtime error.

```python
Traceback (most recent call last):
  File "transformers/examples/rag/finetune_rag.py", line 512, in <module>
    main()
  File "transformers/examples/rag/finetune_rag.py", line 507, in main
    trainer.test()
  File "/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
  File "/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 763, in __test_using_best_weights
    results = self.fit(model)
  File "/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 445, in fit
    results = self.accelerator_backend.train()
  File "/python3.6/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 148, in train
    results = self.ddp_train(process_idx=self.task_idx, model=model)
  File "/lib/python3.6/site-packages/pytorch_lightning/accelerators/ddp_accelerator.py", line 269, in ddp_train
    model = self.trainer.precision_connector.connect(model)
  File "/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/precision_connector.py", line 78, in connect
    model, optimizers = self.backend.connect(model, self.trainer.optimizers)
  File "/lib/python3.6/site-packages/pytorch_lightning/plugins/apex.py", line 37, in connect
    model, optimizers = self.configure_apex(amp, model, optimizers, self.trainer.amp_level)
  File "/lib/python3.6/site-packages/pytorch_lightning/plugins/apex.py", line 102, in configure_apex
    model, optimizers = amp.initialize(model, optimizers, opt_level=amp_level)
  File "/lib/python3.6/site-packages/apex/amp/frontend.py", line 358, in initialize
    return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
  File "/lib/python3.6/site-packages/apex/amp/_initialize.py", line 171, in _initialize
    check_params_fp32(models)
  File "/lib/python3.6/site-packages/apex/amp/_initialize.py", line 87, in check_params_fp32
    name, param.type()))
  File "/lib/python3.6/site-packages/apex/amp/_amp_state.py", line 32, in warn_or_err
    raise RuntimeError(msg)
RuntimeError: Found param model.rag.question_encoder.question_encoder.bert_model.embeddings.word_embeddings.weight with type torch.cuda.HalfTensor, expected torch.cuda.FloatTensor.
When using amp.initialize, you do not need to call .half() on your model
before passing it, no matter what optimization level you choose.
```
To add, I am using Nvidia's Apex library for --fp16 training.

## Expected behavior
fine-tuning to complete, with generated models in the output directory.

Have spent a couple hours tinkering with no clue how to proceed, any help would be appreciated. Thank you!
I cannot really reproduce when just running the fine-tuning script @ritvik1512 could you provide us maybe with a complete code example to reproduce the error? A short colab would also be very helpful! 

Also pinging @lhoestq here since he knows more about RAG fine-tuning<|||||>I haven't experienced issues using --fp16 for apex. A code example to reproduce the error would be welcome indeed<|||||>Thanks for the quick response!
I used the following parameters while running `finetune.py`

```python
python finetune_rag.py \
    --data_dir $DATA_DIR \
    --output_dir $OUTPUT_DIR \
    --model_name_or_path $MODEL_NAME_OR_PATH \ #(facebook/rag-sequence-base)
    --model_type rag_sequence \
    --gpus 4 \
    --fp16 \
    --index_name custom \
    --passages_path $PASSAGE_PATH \ #(an extremely short knowledge source with 2 entries for test)
    --index_path $INDEX_PATH \ #(corresponding index)
    --do_predict \
    --do_train \
    --n_val -1 \
    --val_check_interval 0.25 \
    --train_batch_size 4 \ #(reduced from 8 to avoid OOM)
    --eval_batch_size 1 \
    --max_source_length 128 \
    --max_target_length 25 \
    --val_max_target_length 25 \
    --test_max_target_length 25 \
    --label_smoothing 0.1 \
    --dropout 0.1 \
    --attention_dropout 0.1 \
    --weight_decay 0.001 \
    --adam_epsilon 1e-08 \
    --max_grad_norm 0.1 \
    --lr_scheduler polynomial \
    --learning_rate 3e-05 \
    --num_train_epochs 1 \ #(set at 1 for testing)
    --warmup_steps 500 \
    --gradient_accumulation_steps 1
```
I will try putting together a colab example very soon, in the meantime let me know if the above snippet helps, thanks!<|||||>@patrickvonplaten @lhoestq apologies for a delayed response. I did try putting together a Google Colab environment [(here)](https://colab.research.google.com/drive/1LlWS6tWWp1Oo4ygUE_J53bBTzUxWlF6J?usp=sharing) replicating my local and with **no** --fp16 I could not get the training to end, at around 20% of the way `tcmalloc: large alloc` warnings show up and the runtime resets with all RAM used.

Meanwhile using --fp16, the training it fails with an `IndexError: list index out of range` before the training even starts. 

By the way, I follow the exact code as above (with --fp16) on my local machine and while there the training does do through all the epochs it ends up giving out the error mentioned at the top.

Therefore I am kind of at a loss as to where the issue is and how to proceed. 
Please let me know if you see any possible ways out, thanks!<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.