Multi-GPU setting: Expected to mark a variable ready only once (RuntimeError)
### System Info

I’m trying to run a Huggingface model on a multi-GPU environment. The problem is that when I’m processing multiple inputs which are bound to each other from a single class (shared-weights encoder), I’m getting `RuntimeError: Expected to mark a variable ready only once`. While if I use this encoder module only once, for processing the first input, I won’t get this error, in multi-GPU setting. Should add that I don't get this error when training on single GPU. 

To make it clearer, here is the structure (I have simplified the code):

```
class LEDModel():
    
    def __init__(self, ...)
	   self.encoder = ...


    def forward(input_ids, ...):

	   encoder_outputs = self.encoder(input_ids, ...)

	   # filter encoder_outputs and construct another tensor called 'input_ids_selected'

	   encoder_outputs = self.encoder(input_ids_selected, ...)
           ...

	   return LEDSeq2SeqModelOutput(
            last_hidden_state=decoder_outputs.last_hidden_state,
            past_key_values=decoder_outputs.past_key_values,
            sent_scores=sent_scores,
            sect_scores=sect_scores,
            decoder_hidden_states=decoder_outputs.hidden_states,
            encoder_last_hidden_state=encoder_outputs.last_hidden_state,
        )
```

when I remove this line: `encoder_outputs = self.encoder(input_ids_selected, ...)`, I don't run into this error. Should say that to filter `encoder_outputs` from the first pass of the encoder, I’m using other modules (linear layers) to find important `input_ids`, retaining those in input_ids_selected. You can perceive this as a two-step summarizer.



### Who can help?

@patrickvonplaten @ydshieh @patil-suraj

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Having a weight-shared module inside LEDModel() class will reproduce this error. 

### Expected behavior

Currently, I'm getting `Expected to mark a variable ready only once (RuntimeError)` **in multi-GPU** configuration. I expect to run this model flawlessly in this setting. 
Hi @sajastu Could you share a (minimal) code snippet that could reproduce this issue? Thank you.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.