Can't Run UL2
### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: I guess I tried :)

### Who can help?

@patrickvonplaten , led porting UL2.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
with init_empty_weights():
    model = OPTModel.from_pretrained("facebook/opt-30b", device_map="auto", offload_folder='./offload_folder').to('cuda')
```
Error message
```python
Exception has occurred: ValueError
weight is on the meta device, we need a `value` to put in on 1.
  File "/media/ntu/volume1/home/s121md302_06/workspace/code/yalb/ul2_test.py", line 25, in <module>
    model = T5ForConditionalGeneration.from_pretrained("google/ul2", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16, device_map='auto').to('cuda')
```
![image](https://user-images.githubusercontent.com/45140242/179469229-5784bc16-9405-489d-b493-838b7f290710.png)


### Expected behavior

I expect the model to be successfully loaded with sharded parameters.
Looks like lm_head.weight is causing the problem. But I thought this item was in _keys_to_ignore_on_load_missing of T5ForConditionalGeneration. How could this happen?<|||||>google/ul2 · Splitting the model of multiple GPU's
https://huggingface.co/google/ul2/discussions/4
