parallelize and deparallelize method for GPT-Neo series model
# 🚀 Feature request

Parallelize and deparallelize methods for distribution of attention modules across multiple GPUs.

## Motivation
Finetuning GPT Neo 2.7B model on 12 GB GPU gives out of memory error. Having a parallelize method would allow us to train that model by splitting attention modules across multiple GPUs of smaller VRAM.

## Your contribution
Considering [this line](https://github.com/huggingface/transformers/blob/daf0d6a97bb0225a2571a2612b8285e2c3913992/src/transformers/models/gpt2/modeling_gpt2.py#L522) in GPT2 code and the absence of doc for parallelize method in [GPT2 documentation](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2model), wanted to know if these methods are still supported. If not, what is the recommended method for fine-tuning large transformer models like GPT-Neo?

If they are still supported, I can take up this task and submit PR for both methods as well as documentation fix.




This is answered in #11054.

(I'm in a similar situation as you. I'm just going to go with the suggestion and use DeepSpeed instead of model parallelism.)<|||||>Thanks, didn't saw that. Parallelism notes are also awesome.