Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
## Environment info

- `transformers` version: 3.1.0
- Platform: ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?): 1.14.0
- Using GPU in script?:  Yes
- Using distributed or parallel set-up in script?: No 

## Information

Model I am using: XLM

The problem arises when using:
When stack two transformes on top of each other and run them on GPUs. (Note that on CPU the following code works fine)

## To reproduce
Steps to reproduce the behavior:

```
import torch
from torch import nn
from transformers import AutoConfig, AutoModel

class Model(nn.Module):

    def __init__(self):

        super().__init__()
        config = AutoConfig.from_pretrained('xlm-mlm-tlm-xnli15-1024')
        self.transformer1 = AutoModel.from_config(config)
        self.transformer2 = AutoModel.from_config(config)
       

    def forward(
            self,
            input_ids=None,
            input_embeds=None,
    ):
        
        outputs = self.transformer1(input_ids)
        outputs = self.transformer2(inputs_embeds=outputs[0])
        return outputs
    
device = "cuda" if torch.cuda.is_available() else "cpu"
model = Model().to(device)

inps = torch.randint(1,256,[1,256],device=device)

model(input_ids=inps)

```

The stack trace:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-35-5ada78c6ff36> in <module>
     34 inps = torch.randint(1, 256, [1, 256], device=device)
     35 
---> 36 model(input_ids=inps)

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-35-5ada78c6ff36> in forward(self, input_ids, input_embeds)
     26         print(outputs[0].is_cuda)
     27         outputs = outputs[0].to(input_ids.device)
---> 28         outputs = self.transformer2(inputs_embeds=outputs)
     29         return outputs
     30 

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.8/site-packages/transformers/modeling_xlm.py in forward(self, input_ids, attention_mask, langs, token_type_ids, position_ids, lengths, cache, head_mask, inputs_embeds)
    519         # if src_enc is not None:
    520         #     assert self.is_decoder
--> 521         #     assert src_enc.size(0) == bs
    522 
    523         # generate masks

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
