Trouble parallelizing GPT-NeoX
### System Info

```shell
- `transformers` version: 4.20.0.dev0
- Platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.6.0
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes
```


### Who can help?

@patil-suraj, @patrickvonplaten, @LysandreJik, @stas00

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I have been trying to implement the `.parallelize()` method for GPT-NeoX. I am aware that it is soon going to be obsolete, but I've been having trouble getting `accelerate` working [with my custom codebase](https://github.com/bigscience-workshop/lm-evaluation-harness/tree/cjlovering/accelerate-2), but `.parallelize()` does work so I figured I would give it a shot. My fork can be found [here](https://github.com/StellaAthena/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py) and is based off the implementation for GPT-J.

Unfortunately, it does not seem like I have implemented it correctly. That said, the error message I am getting seems quite strange to me and is not what I would expect to get. I have verified that my code runs correctly for GPT-J-6B, including parallelism.

```
Traceback (most recent call last):
  File "/home/mchorse/bigbio/lm-evaluation-harness/test.py", line 4, in <module>
    model.parallelize()
  File "/home/mchorse/miniconda3/envs/evalharness/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1185, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'GPTNeoXForCausalLM' object has no attribute 'parallelize'
```

If you want to run my code, you can do so by following the instructions [here](https://github.com/bigscience-workshop/lm-evaluation-harness#overview). However I do not recommend doing so. Instead, the same error can be generated by running the following basic script:
```python
from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast

model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b")
model.parallelize()
tokenizer = GPTNeoXTokenizerFast.from_pretrained("EleutherAI/gpt-neox-20b")

prompt = "GPTNeoX20B is a 20B-parameter autoregressive Transformer model developed by EleutherAI."

input_ids = tokenizer(prompt, return_tensors="pt").input_ids

gen_tokens = model.generate(
    input_ids,
    do_sample=True,
    temperature=0.9,
    max_length=100,
)
gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

### Expected behavior

I expect my code to work.
Gently pinging @patil-suraj and maybe also @sgugger regarding `accelerate` here - I think `accelerate` should be used here rather than `parallelize` no? 

@LysandreJik @sgugger should we maybe fully deprecate `parallelize()` now?<|||||>Yes the `parallelize` API will be fully deprecated soon (like this week or the next) so there is no point adding support to new models.

> I've been having trouble getting accelerate working [with my custom codebase](https://github.com/bigscience-workshop/lm-evaluation-harness/tree/cjlovering/accelerate-2)

Could you tell us more about the problem you are encountering there maybe?<|||||>Hi @sgugger , I want to do inference on t5-11b model and tried the `parallelize` method. It showed the error like this way

```
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 683, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 950, in _test_impl
    results = self._run(model, ckpt_path=self.tested_ckpt_path)
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run
    self._dispatch()
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1271, in _dispatch
    self.training_type_plugin.start_evaluating(self)
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 178, in start_evaluating
    self.spawn(self.new_process, trainer, self.mp_queue, return_result=False)
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 201, in spawn
    mp.spawn(self._wrapped_function, args=(function, args, kwargs, return_queue), nprocs=self.num_processes)
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/zeyi/.conda/envs/lr_dataset/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 136, in join
    signal_name=name
torch.multiprocessing.spawn.ProcessExitedException: process 2 terminated with signal SIGABRT
python-BaseException
wandb: Waiting for W&B process to finish... (success).
wandb:                                                                                
wandb: Synced lrgenerative_logic_comp1_v7_1.0_new_seed42_trim_filtered_t5_11b_13_06_2022_ddd9ce1c: https://wandb.ai/soumya_research/lr_dataset/runs/32ujsgo3
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20220613_020333-32ujsgo3/logs
[W CudaIPCTypes.cpp:21] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
```

I tried to find some solutions and also set the `num_workers` of dataset class to 0. But still doesn't work.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>@ZeyiLiao,

please don't use `parallelize` for the `t5-11b` model instead you can load it using the `device_map="auto"` see [here](https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/model#large-model-loading)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.