Autotokenizer/ LED/BARTTokenizer won't cast to CUDA
### System Info

Hello,

I'm running into the following issue when trying to run an LED model.  It appears that the tokenizer won't cast into CUDA.  I've seen this work in the past, but apparently something has gone amiss.  I'm not entirely sure why this behavior is being exhibited.


`from transformers import (
    AutoTokenizer,
    LEDForConditionalGeneration,
)
import torch
import pickle
 
device = "cuda:0" if torch.cuda.is_available() else "cpu"
primer_path = "allenai/PRIMERA-multixscience"
tokenizer = AutoTokenizer.from_pretrained(primer_path, return_tensors="pt").to(device)
model = LEDForConditionalGeneration.from_pretrained(primer_path).to(device)
model.gradient_checkpointing_enable()
PAD_TOKEN_ID = tokenizer.pad_token_id
DOCSEP_TOKEN_ID = tokenizer.convert_tokens_to_ids("<doc-sep>")`

`2022-09-30 05:10:51 : Traceback (most recent call last):
2022-09-30 05:10:51 :   File "/summarization.py", line 17, in <module>
2022-09-30 05:10:51 :     tokenizer = AutoTokenizer.from_pretrained(primer_path, return_tensors="pt").to(device)
2022-09-30 05:10:51 : AttributeError: 'LEDTokenizerFast' object has no attribute 'to'`

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Simply try to import PRIMERA and use a tokenizer with CUDA

### Expected behavior

I would expect the tokenizer to cast onto the GPU, not give an exception that to is not an attribute.
Hello @M-Chimiste Please assign me and btw I'm new to the contribution so guide accordingly
Thank you!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.