Distributed Support for OPT models in transformers
### Feature request

Hi
Thanks a lot for adding OPT models to transformers. As of now the 55GB 30B parameter model needs to be loaded into a single GPU, otherwise it t[hrows a CUDA memory error](https://discuss.huggingface.co/t/running-inference-on-opt-30m-on-gpu/17895/2).

It would be great if we could have distributed support for these[ models similar to gpt2](https://github.com/huggingface/transformers/pull/7772) so we can leverage multiple GPUs to run them

### Motivation

Make it easier to run OPT models on available infra

### Your contribution

NA, 
can test changes on my system
@patrickvonplaten  rasising a feature request for model parallelism for  the newly added OPT models. Please triage/comment as you see fit<|||||>Same answer as in https://github.com/huggingface/transformers/pull/17245#issuecomment-1128064880 here. Think we'll soon have something that works out of the box cc @sgugger <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.