BART run run_train.sh RuntimeError: expected device cuda:0 but got device cpu
# 🐛 Bug

## Information

Model I am using (BART):

Language I am using the model on CNN dailymail (English, Chinese ...):

The problem arises when using:
* [* ] the official example scripts: (give details below)
text summarization of bart, run the script run_train.sh, following the guideline
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [*] an official GLUE/SQUaD task: (give the name)
the text summarization task, using cnn dailymail dataset.
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. run the script run_train.sh in examples/summarization/bart/run_train.sh
2. got the error information RuntimeError: expected device cuda:0 but got device cpu


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: I pip install transformers two days ago, I am not sure the version.
- Platform: ubuntu 1604
- Python version: 3.6.7
- PyTorch version (GPU?):  1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Traceback:
Traceback (most recent call last):
  File "run_bart_sum.py", line 166, in <module>
    trainer = generic_train(model, args)
  File "******/transformer_base.py", line 304, in generic_train
    trainer.fit(model)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 676, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "******/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "******/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "******/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 341, in ddp_train
    self.run_pretrain_routine(model)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 924, in run_pretrain_routine
    False)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 263, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 418, in evaluation_forward
    output = model(*args)
  File "******/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/overrides/data_parallel.py", line 96, in forward
    output = self.module.validation_step(*inputs[0], **kwargs[0])
  File "******/run_bart_sum.py", line 58, in validation_step
    loss = self._step(batch)
  File "******/run_bart_sum.py", line 44, in _step
    lm_labels=lm_labels.cuda(),
  File "******/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "******/run_bart_sum.py", line 32, in forward
    lm_labels=lm_labels,
  File "******/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 925, in forward
    decoder_cached_states=decoder_cached_states,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 844, in forward
    decoder_cached_states=decoder_cached_states,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 499, in forward
    need_attn_weights=self.output_attentions,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 372, in forward
    attn_mask=attention_mask,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 629, in forward
    attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask
RuntimeError: expected device cuda:0 but got device cpu

I meet the same bug when i use bart as an embedding layer.
Have u solve the problem?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> I meet the same bug when i use bart as an embedding layer.
> Have u solve the problem?

any luck on this?  seeing the same "RuntimeError: expected device cuda:0 but got device cpu"