RuntimeError with DistributedDataParallel 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Ubuntu 18.04
- Python version: 3.7.7.final.0
- PyTorch version (GPU?): 1.6.0 gpu
- Tensorflow version (GPU?): X
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed set-up
- Conda : 4.8.3 
- cuda =11.0

### Who can help
 @LysandreJik  @sgugger

## Information

Model I am using (Bert, XLNet ...): bert mulitilingual cased

The problem arises when using:
* [x] my own modified scripts : Triplet Loss applied to sentence similarity, to practice writing efficient distributed learning code.


The tasks I am working on is:
* [x] my own task or dataset: Triplet Loss applied to sentence similarity, to practice writing efficient distributed learning code.


## To reproduce

I have not been able to generte a minimal version yet.

Stack trace

```
Traceback (most recent call last):
[...]
  File "my_program.py", line 246, in epoch
    loss.backward()
  File "conda/lib/python3.7/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 200]] is at version 9; expected version 7 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
```

My model : 

```python

def __init__(self, trainfile, devfile, outdir, rank=0):
         # default config
          self.max_sent_len = 200
          self.batchsize = 10
          self.lr = 2e-5
          self.accumulated_batches = 6
          self.max_epochs = 10
          self.seed = 42
          self.embs = "bert-base-multilingual-cased"
          self.tokenizer, self.model = self.transformers(self.embs)
          self.tripletloss = lossless_triplet_loss.LosslessTripletLoss()
  
          print("use %s gpus" % torch.cuda.device_count())
          self.model = self.model.to(rank)
          self.model = torch.nn.parallel.DistributedDataParallel(self.model, device_ids=[rank], output_device=rank, find_unused_para  meters=True)
          self.rank = rank
  
          half = False
          self.model = self.model.to(self.rank) 
  
          self.optimizer = optim.Adam(filter(lambda param: param.requires_grad, self.model.parameters()), lr=self.lr)
  
          self.traindataset = data.TripleDataset2(trainfile, self.tokenizer, self.max_sent_len)
          self.devdataset = data.TripleDataset2(devfile, self.tokenizer, self.max_sent_len)
          self.traindataloader = DataLoader(self.traindataset,
                                            collate_fn=self.collate_fn,
                                            batch_size = self.batchsize,
                                            num_workers = 0)
          self.devdataloader = DataLoader(self.devdataset,
                                          collate_fn=self.collate_fn,
                                          batch_size = self.batchsize,
                                         num_workers = 0)

 def train(self, epoch):
          self.model.train()
  
          iterations = int(math.ceil(len(self.traindataset)/self.batchsize))
          iterator = tqdm.tqdm(enumerate(self.traindataloader),
                               ncols=100,
                               ascii=True,
                               desc="epoch %d" % (epoch),
                               mininterval=1.0,
                               total = iterations)
          lastloss = None
          for batch_idx, anchor_pos_neg in iterator:
              (anchor, amask), (pos, pmask), (neg, nmask) = anchor_pos_neg
  
              anchor = anchor.to(self.rank)
              pos = pos.to(self.rank)
              neg = neg.to(self.rank)
  
              amask = amask.to(self.rank)
              pmask = pmask.to(self.rank)
              nmask = nmask.to(self.rank)
  
              anchor_vec = self.tokens2vec(anchor, attention_mask=amask)
              pos_vec = self.tokens2vec(pos, attention_mask=pmask)
              neg_vec = self.tokens2vec(neg, attention_mask=nmask)
              loss, distp, distn = self.calc_triplet_loss(anchor_vec, pos_vec, neg_vec)
  
              loss.backward()
              if (batch_idx+1) % self.accumulated_batches == 0:
                  self.optimizer.step()
                  self.optimizer.zero_grad()

      def tokens2vec(self, tokens_tensor, attention_mask=None):
          hidden_states = self.model(tokens_tensor, attention_mask=attention_mask)[0]
          token_vecs = torch.squeeze(hidden_states, dim=0)
  
          a = torch.sum(torch.mul(token_vecs, torch.unsqueeze(attention_mask, 2)), dim=1)
          m = torch.unsqueeze(torch.sum(attention_mask, dim=1), 1)
          sentence_embedding = torch.sigmoid(torch.div(a, m))
          return sentence_embedding
  
  
      def transformers(self, name):
          if name.startswith("bert"):
              from transformers import BertTokenizer, BertModel
              tokenizer = BertTokenizer.from_pretrained(name, do_lower_case=False)
              model = BertModel.from_pretrained(name)
          else:
              tokenizer = None
              model = None
              print("ERROR: invalid embeddings name <%s>" % name, file=sys.stderr)
          return tokenizer, model

```

The error happens with batch_idx = 0.

200 is my maximum sequence length. I only use Longtensor as input to BertModel (for anchor, pos and neg). 
What I find strange is that I am sending samples in batches of size 10, so I am wondering if my problem is really with the embedding layer, or rather caused by my custom collate_fn  : 

```python
  def collate_fn(self, batch):
          anchor = []
          pos = []
          neg = []
          anchor_mask = []
          pos_mask = []
          neg_mask = []
          for a,p,n in batch:
              anchor.append(a[0])
              pos.append(p[0])
              neg.append(n[0])
              anchor_mask.append(a[1])
              pos_mask.append(p[1])
              neg_mask.append(n[1])
          return (torch.stack(anchor, dim=0),torch.stack(anchor_mask, dim=0)), \
               (torch.stack(pos, dim=0),torch.stack(pos_mask, dim=0)), \
               (torch.stack(neg, dim=0),torch.stack(neg_mask, dim=0))
```

My model is working well with only one GPU, or when using DataParallel. My issue arises only with DistributedDataParallel.


## Expected behavior
A running training.
The `BertModel` runs absolutely fine in `DistributedDataParallel` (just retried) so the issue probably comes from the rest of your code and not the library. I can't see the code for the `LosslessTripletLoss`, maybe the issues comes from there.

In any case, to debug I would follow the instruction given in the error message:
```
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 200]] is at version 9; expected version 7 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
```<|||||>Well, thanks for your tips. 

But when using set_detect_anomaly, my problem seems to be located in EmbeddingBackward :

```
[W python_anomaly_mode.cpp:60] Warning: Error detected in EmbeddingBackward. Traceback of forward call that caused the error:
  File "<string>", line 1, in <module>
  File "/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/lib/python3.7/multiprocessing/spawn.py", line 118, in _main
    return self._bootstrap()
  File "/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "my_program.py", line 525, in demo_basic
    f4n.train()
  File "my_program.py", line 213, in train
    self.epoch(epoch)
  File "my_program.py", line 249, in epoch
    pos_vec = self.tokens2vec(pos, attention_mask=pmask)
  File "my_program.py", line 444, in tokens2vec
    hidden_states = self.model(tokens_tensor, attention_mask=attention_mask)[0]
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 511, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/transformers/modeling_bert.py", line 831, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/transformers/modeling_bert.py", line 198, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
```



The problem occurs when calling BertModel the second time in the epoch, when calculating "pos".<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi, have you solved this problem? I also met this issue.<|||||>No, sorry, I finally gave up.<|||||>Same Error. Any one has solved the problem?<|||||>I've solved this problem. Mine was because when I build the input sequences as follows:
`decoder_inputs = labels.narrow(1, 0, seqlen - 1).clone()`
`decoder_masks = label_masks.narrow(1, 0, seqlen - 1).clone()`
`decoder_labels = labels.narrow(1, 1, seqlen-1).clone()`
`decoder_label_masks = label_masks.narrow(1, 1, seqlen - 1).clone()`
`decoder_labels[~decoder_label_masks.bool()] = -100`

I forgot to add `.clone()` at the beginning. So when I change some of the `decoder_labels` to be -100, some of the elements in `decoder_inputs` were also changed, which caused the error.

Hope this information could help you a bit!<|||||>> I've solved this problem. Mine was because when I build the input sequences as follows:
> `decoder_inputs = labels.narrow(1, 0, seqlen - 1).clone()`
> `decoder_masks = label_masks.narrow(1, 0, seqlen - 1).clone()`
> `decoder_labels = labels.narrow(1, 1, seqlen-1).clone()`
> `decoder_label_masks = label_masks.narrow(1, 1, seqlen - 1).clone()`
> `decoder_labels[~decoder_label_masks.bool()] = -100`
> 
> I forgot to add `.clone()` at the beginning. So when I change some of the `decoder_labels` to be -100, some of the elements in `decoder_inputs` were also changed, which caused the error.
> 
> Hope this information could help you a bit!

Thanks a lot!<|||||>I met exact same issue with triplet loss using transformer 4.5.1 in NCCL distributed training setting.
I found this problem may occur to using multiple forward Bert output and compute loss at once.
I solved this by concat inputs ahead and chunk it to calculate loss:
```python
input_ids = torch.cat([anchor_input_ids, pos_input_ids, neg_input_ids], dim=0)
attention_mask = torch.cat([anchor_attention_mask, pos_attention_mask, neg_attention_mask], dim=0)
token_type_ids = torch.cat([anchor_token_type_ids, pos_token_type_ids, neg_token_type_ids], dim=0)

emb = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
anchor_emb, pos_emb, neg_emb = emb.pooled_output.chunk(3, dim=0)
loss = criterion(anchor_emb, anchor_emb, anchor_emb)
```
<|||||>My problem was solved after adding `broadcast_buffers=False` to `torch.nn.parallel.DistributedDataParallel`
Following https://github.com/ashkamath/mdetr/issues/16#issuecomment-878388469