Add model parallelism to CodeGen
This PR adds model parallelisim to the CodeGen model. I have been using this since August.
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20016). All of your documentation changes will be reflected on that endpoint.<|||||>ok, I did not know that. Thanks for the info!

Does `device_map="auto"` also work for CodeGen?<|||||>Yes, it's supported (at least on the main branch)!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>> Thanks for your PR but this code for model parallelism is deprecated and will be removed from other code files. To use parallelism, please load the model with `device_map="auto"`.

I'm a bit confused about model parallelism in Huggingface. I'm trying to fine-tune a CodeGen model using Huggingface Trainer. Is loading the model with `device_map="auto"` the right way to enable model parallelism?

From what I read [here](https://huggingface.co/docs/transformers/v4.28.1/perf_train_gpu_many#naive-model-parallelism-vertical-and-pipeline-parallelism), model parallelism is only supported by GPT2 and T5.