Enable to pass two or model pretrained models to the Trainer.
### Feature request

I tried a compound architecture that uses two tower models that get initialized with from_pretrained methods. 


Then I wrapped the Custom model class with the Pretrained model class. 

It works well in a single GPU.  But fails in multiple GPU settings. 


### Motivation

It would be useful to change models like DPR.  

### Your contribution

I have tested a few methods and can help to improve. 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.