RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)
## Environment info

- `transformers` version: 4.12.5
- Platform: Linux-5.10.90+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.1 (True)
- Tensorflow version (GPU?): 2.6.2 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik @vanpelt  @arfon 


## Information
I am using Captum for interpreting the attributions of the tokens in each layer using Layer-Conductance
`lc = LayerConductance(predict, model.bert.encoder.layer[i])`

Now, in the line
`layer_attributions = lc.attribute(inputs=input_ids, baselines=ref_input_ids, additional_forward_args=(attention_mask,))`
RuntimeError is generating.

A helper function to perform forward pass of the model and make predictions.
`def predict(input_ids, attention_mask=None):
    outputs, attention_weights = model(input_ids=input_ids, attention_mask=attention_mask)
    preds = torch.softmax(outputs, dim = 1)[0][1].unsqueeze(0)
    return preds`

Model I am using: "google/muril-base-cased"


#2952 

![2](https://user-images.githubusercontent.com/20766288/150690257-d4f92158-6709-4728-b8bb-e4ac995059b3.png)
![1](https://user-images.githubusercontent.com/20766288/150690261-3f31369f-3e5d-49a6-83dc-e8e9e9f311d8.png)



## Expected behavior
Code is working fine during training and prediction but raising errors while interpreting the layers with captum.

Any help would be greatly appreciated.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@neeraj1909 I am facing the same issue, did you manage to figure this out by any chance?