Problem of training gpt-neox from transformer 
### System Info

```
PyTorch
Transformer

2x A100 (40 GB)
```




### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I couldn't fine tune the gpt-neox 20b from transformer library. Also, there is no official documentation how to train or fine tune the LLM with ease with transformer library. 

https://huggingface.co/EleutherAI/gpt-neox-20b

### Expected behavior

It should run.
Maybe you should try getting help on the [forums](https://discuss.huggingface.co/)? We only use GitHub issues for bugs and feature requests only, and I don't think this is relevant here.<|||||>@sgugger Unfortunately this forum is not much helpful, you can search with 'gpt-neox' word, and you'd find many questions without response. Not blaming, but I like to know if transformer support training with this large LM model or just the inference? The documentaiton is missing for this model, or I've missed! 

<|||||>Transformers supports training and inference of all its models, but you will probably need to use other libraries like DeepSpeed or PyTorch FSDP to do such training depending on your hardware. Since this is a model that will 80GB in float32 by itself, it's very unlikely you will be able to train it on the hardware your provided at the top. Training those models from scratch require a lot more GPU compute (which is also why there are fewer tutorials available as it doesn't touch as many people as smaller models).

You might have more luck fine-tuning the exisiting model with the [peft library](https://github.com/huggingface/peft). Otherwise this [doc page](https://huggingface.co/docs/transformers/perf_train_gpu_many) will list resources you can use.