VisualBERT - ModuleAttributeError
## Environment info
- `transformers` version: 4.9.1
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.9.0
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.5.0-rc1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help
@gchhablani

## Information

I am using the recent VisualBERT model.  

When giving the inputs to model, a ModuleAttributeError occurs, since internally the class VisualBertEmbeddings calls self.input_embeds but that class does not have that attribute (e.g., in the _init_), thus the error.  

`class VisualBertEmbeddings(nn.Module):`
(...)
`token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.input_embeds.device)` 
(self.input_embeds does not exist before this line) 

The problem arises when using:
* the official example scripts: 
```
from transformers import BertTokenizer, VisualBertModel
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = VisualBertModel.from_pretrained('uclanlp/visualbert-vqa-coco-pre')
inputs = tokenizer("The capital of France is Paris.", return_tensors="pt")
visual_embeds =  torch.zeros((1,36,2048)) #example of ROI features
visual_token_type_ids = torch.ones(visual_embeds.shape[:-1], dtype=torch.long) #example
visual_attention_mask = torch.ones(visual_embeds.shape[:-1], dtype=torch.float)
inputs.update({{
    "visual_embeds": visual_embeds,
    "visual_token_type_ids": visual_token_type_ids,
    "visual_attention_mask": visual_attention_mask
}})
outputs = model(**inputs)
```

## To reproduce

Steps to reproduce the behavior:

1. Follow the official example script

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/models/visual_bert/modeling_visual_bert.py", line 807, in forward
    embedding_output = self.embeddings(
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/models/visual_bert/modeling_visual_bert.py", line 126, in forward
    token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.input_embeds.device)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 778, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'VisualBertEmbeddings' object has no attribute 'input_embeds
```

## Expected behavior

The class VisualBertEmbeddings should have self.input_embeds before calling it, otherwise the VisualBert will not work, since internally there is that bug.   


Thank you in advance for your help!

There seem to be a bug in the `VisualBertEmbeddings` class indeed. Mind opening a PR? You can probably just replace `self.input_embeds.device` by `self.position_ids.device`.<|||||>Great, thank you for looking and suggesting the fix! I would not mind, but I will go on vacation very soon, so I won’t be able to follow up on this topic.  <|||||>@RitaRamo Thanks for trying it out. Sorry for the late response.
@NielsRogge Thanks for suggesting a fix. I'll do it asap. <|||||>Fixed in #13017 