one of the variables needed for gradient computation has been modified by an inplace operation
### System Info

* Ubuntu 20.04
* Architecture x86_64
* 3 x Tesla P100-PCIE-12GB
* Python 3.8.10
* torch==1.12.1+cu116

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I encountered the error `one of the variables needed for gradient computation has been modified by an inplace operation...` when training my model with DistributedDataParallel (DDP). My code run smoothly when I do not use DDP. I have spent time inspecting the problem and below is the minimal code for reproducing the problem.
```python
import torch
from torch import nn
import argparse


class BertEmbeddings(nn.Module):

    def __init__(self, config):
        super().__init__()
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)))

    def forward(
        self, input_ids, past_key_values_length=0
    ):
        seq_length = input_ids.shape[1]
        position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]
        return self.position_embeddings(position_ids)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--local_rank", type=int, default=-1)
    args = parser.parse_args()

    local_rank = args.local_rank
    torch.cuda.set_device(local_rank)
    device = torch.device("cuda", local_rank)
    torch.distributed.init_process_group(backend="nccl")

    w = BertEmbeddings(config=argparse.Namespace(max_position_embeddings=10, hidden_size=24))
    w.to(device)

    # setup distributed
    w = torch.nn.parallel.DistributedDataParallel(w, device_ids=[local_rank],
                                                    output_device=local_rank,
                                                    find_unused_parameters=False)

    input_ids = torch.tensor([[1, 2, 3]]).to(device)
    x = w(input_ids)
    y = w(input_ids)
    M = torch.sum(x)
    M.backward()


if __name__ == "__main__":
    main()

```

Suppose this code is put in a file named `debug_distributed.py`. I run this code with the command
```shell
python -m torch.distributed.launch --nproc_per_node=3 debug_distributed.py
```

, and I got the error
<pre>
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 3]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</pre>

If I do not use DDP, there is no such error. Specifically, put the following in a file named `debug_normal.py` and run `python debug_normal.py`
```python
import torch
from torch import nn
import argparse


class BertEmbeddings(nn.Module):

    def __init__(self, config):
        super().__init__()
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)))

    def forward(
        self, input_ids, past_key_values_length=0
    ):
        seq_length = input_ids.shape[1]
        position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]
        return self.position_embeddings(position_ids)


def main():
    w = BertEmbeddings(config=argparse.Namespace(max_position_embeddings=10, hidden_size=24))
    w.to("cuda")
    input_ids = torch.tensor([[1, 2, 3]]).to("cuda")
    x = w(input_ids)
    y = w(input_ids)
    M = torch.sum(x)
    M.backward()


if __name__ == "__main__":
    main()

```

This problem prevents me from training my BertModel in distributed mode. I found that the problem lies on the line `position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]`. It seems like an "inplace  operation" as the error suggests. If I change that line to `position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length].clone()`, the problem will be gone.

I think this problem is much more related to PyTorch. It may be a Pytorch bug. However, the simplest workaround is to add a `.clone()` as I showed above. Currently, `transformers` of version `>=4` uses this "inplace operation" and all `>=4` versions of `transformers` will get this error. So, is there anyway to better fix the problem, so I don't need to change library (`transformers`) code?

### Expected behavior

BertModel works in distributed training with DistributedDataParallel
cc @pacman100 <|||||>Hi @levuloihust99 
do you face the same issue by setting:
```python
find_unused_parameters=True
```<|||||>> Hi @levuloihust99 do you face the same issue by setting:
> 
> ```python
> find_unused_parameters=True
> ```

Setting `find_unused_parameters=True` gave me the exact same error. Additionally, in my example code, it is more performant to set `find_unused_parameters=False` since there is no unused parameters.