Parallelization for OPT model
### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.10.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Lyandrejik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import OPTForCausalLM

model = OPTForCausalLM.from_pretrained("facebook/opt-6.7b").cuda()
model.parallelize()

### Expected behavior

Hi,
I am loading the OPT model. Due to many parameters, I cannot put the whole model on one CUDA device. Therefore, I tried to parallelize the OPT like T5, using "model.parallelize()". However, I find that there is no parallelization mechanism for OPT model now.
Can you please implement the parallelization mechanism for OPT models? Or what can I do to load and run a large OPT model?
I have the same question since directly tuning models with this size (OPT) on one single device is infeasible. It would be great to have the parallelization mechanism implemented for OPT models. Many thanks! <|||||>See this conversation which might help out: https://huggingface.co/facebook/opt-66b/discussions/6<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.