Is AMD supported for transformers and text generation?
### System Info

Hello,

I would like to ask if it is possible to run models like GPT-J and OPT-6.7b using an AMD GPU like RX 6800 16GB.

Specifically using `AutoModelForCausalLM.from_pretrained`. I have similar models (like OPT-1.3B) working on an NVIDIA GPU with half precision, but I don't know if it would seamlessly work on a different GPU brand.

The operating system is Ubuntu.

Thank you.

### Who can help?

@patil-suraj @patrickvonplaten @Narsil @gante_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Example script

```
model = AutoModelForCausalLM.from_pretrained("/home/me/models/opt-1.3b/", torch_dtype=torch.float16).cuda()
tokenizer = AutoTokenizer.from_pretrained("/home/me/models/opt-1.3b/")

input_text = f"Hello"
input_ids = tokenizer.encode(str(input_text), return_tensors='pt').cuda()

output = model.generate(
    input_ids,
    do_sample=True,
    max_length=200,
    temperature=0.9,
).cuda()

reply = tokenizer.decode(output[0], skip_special_tokens=True)
```

### Expected behavior

Working on an AMD GPU (RX 6800 16GB) seamlessly
Hey @oobabooga, are you getting an error when running this code currently on AMD? Could you maybe just try it out and add a stack trace here if it doesn't work? :-) 

Thanks!<|||||>Hello @patrickvonplaten, I am not getting an error, I would just like to know if transformers also works on AMD or if it is exclusive to NVIDIA. I have not found this information anywhere.<|||||>Hey @oobabooga, we're not exclusive to either NVIDIA or AMD, but we do rely on several backends to run transformer models: either PyTorch, TensorFlow, or JAX.

If you setup either of those 3 to run with AMD GPUs, it should run with transformer models without issue.<|||||>Thank you for the clarification, @LysandreJik. I am closing the issue as that answers the question.<|||||>> If you setup either of those 3 to run with AMD GPUs, it should run with transformer models without issue.

Can you please link some resources on how to do that?