model.parallelize() for OPT models 
### Feature request

It would be great if we can have function for fitting OPT model on multiple GPUs using **model.parallelize()**, similar to what we already have for [GPT-J](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/gptj/modeling_gptj.py#L494)

### Motivation

It would be extremely helpful for fitting large OPT models such as opt-30b, opt-13b. An ideal solution would be similar to what has been integrated in [GPT-J](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/gptj/modeling_gptj.py#L494), where we can simply call **model.parallelize()** to load the model on multiple gpus.

### Your contribution

glad to help on making this feature possible
@sgugger can share more on that, but we now recommend leveraging `accelerate` in order to parallelize models. See [tweet](https://twitter.com/huggingface/status/1524783489593360385) that contains a [colab example](https://colab.research.google.com/drive/14wnxMvD9zsiBQo2FtTpxn6w2cpXCcb-7#scrollTo=XUlpcU3iQNhu&uniqifier=1).<|||||>On Transformers main branch, you can also load the OPT models on several GPUs directly with `AutoModel.from_pretrained(checkpoint, device_map="auto")` (or pass along you own device map).

The old `parallelize` API will be deprecated soon.<|||||>@sgugger Hi, that's very helpful for me. However I wonder what's the difference between directly using ```AutoModel.from_pretrained(checkpoint, device_map="auto")```  and that using ```load_checkpoint_and_dispatch()``` and ```infer_auto_device_map()```? Does both of them support fine-tuning something like OPT-30B on multiple GPUs with model parallelism? Thanks a lot for your help!<|||||>No this is inference only. For training/fine-tuning we recommend the use of DeepSpeed Zero-3.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.