load_in_8bit=True crashes GPT-J when running model.generate()
### System Info

Colab Pro, NVIDIA P100
Transformers: 4.22.1
Accelerate: 0.12.0

- `transformers` version: 4.22.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.8.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N



### Who can help?

@patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This works fine, loads the model using 6GB GPU memory, 10GB free
```
from transformers import GPTJForCausalLM, AutoTokenizer
import torch

model =  GPTJForCausalLM.from_pretrained(
    "EleutherAI/gpt-j-6B", 
    revision="float16", 
    torch_dtype=torch.float16, 
    load_in_8bit=True, 
    device_map='auto', 
    low_cpu_mem_usage=True
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")
```
However when I run:
```
prompt = "I "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

gen_tokens = model.generate(input_ids, do_sample=True, temperature=0.9, max_length=5,)
gen_text = tokenizer.batch_decode(gen_tokens)[0]
print(gen_text)
```
The colab environment crashes with an unknown error, during model.generate()

`max_length=5` and a short prompt were tested to lower mem requirements, but this happens with any length and prompt.
Presumbaly it's not a memory issue as I can just about use inference without load_in_8bit, on the same arch.

### Expected behavior

Model generates exciting tokens, decodes, and prints!
Hey @petertjmills -- can you share with us the error you're seeing or, better yet, a link to a colab with the error?<|||||>Thank you for the reply! I threw together a quick notebook to send and it worked 🙄

I've found the source of the issue. In the bitsandbytes readme they specify:
```
Hardware requirements:

    LLM.int8(): NVIDIA Turing (RTX 20xx; T4) or Ampere GPU (RTX 30xx; A4-A100); (a GPU from 2018 or older).
    8-bit optimizers and quantization: NVIDIA Maxwell GPU or newer (>=GTX 9XX).
```

The first time I got a P100 (Pascal arch, 2016)
The second time I got a T4, go figure. 
May be worth documenting?<|||||>> May be worth documenting?

Definitely! Would you like to open a PR with a more informative exception? :D