bnb_4bit for Flan-T5-XL/XXL?  Can't load on Colab T4...
### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger (I think)

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Not sure this is the best place to raise the issue - please feel free to redirect.

Following your great [Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA blog](https://huggingface.co/blog/4bit-transformers-bitsandbytes), I used your example [Basic usage Google Colab notebook](https://colab.research.google.com/drive/1ge2F1QSK8Q7h0hn3YKuBCOAS0bK8E0wf?usp=sharing), modified the model_id to "google/flan-t5-xl" (the 3B model) as follows.  

However, the Colab session crashes (free T4 GPU).
According to the blog, T5 is supported, but I may be missing something.  Please point me the right direction?

```
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from transformers import T5ForConditionalGeneration

model_id = "google/flan-t5-xl"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16import torch

)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model_4bit = T5ForConditionalGeneration.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")
# crashes - runs out of memory
```

### Expected behavior

I had expected the model to load and use test it.  I'd like to finetune Flan-T5 using LoRA.  In case the settings for T5 needs to be something specific, it would be helpful to know what they are.  Thanks.
Can load using b16 sharded version.