Why is tokenized-text Non-Fast when it is of type BatchEncoding and the tokenizer is Fast?
### System Info

Huggingface Transformers version **4.21.1**
PyTorch version **1.12.1+cu116**
Python version **3.10.4**
Platform Ubuntu **22.04.1 LTS**

**transformers-cli env**
Traceback (most recent call last):
  File "/home/vin/.local/bin/transformers-cli", line 5, in <module>
    from transformers.commands.transformers_cli import main
  File "/home/vin/.local/lib/python3.10/site-packages/transformers/commands/transformers_cli.py", line 24, in <module>
    from .pt_to_tf import PTtoTFCommand
  File "/home/vin/.local/lib/python3.10/site-packages/transformers/commands/pt_to_tf.py", line 21, in <module>
    from datasets import load_dataset
ModuleNotFoundError: No module named 'datasets'


### Who can help?

@LysandreJik, @SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### Step 1: Text is tokenized as follows:
```
        batch_nnIn_ids = self.tokenizer(text=batch_histories,
                                        text_pair=batch_user_input_pretok,
                                        is_split_into_words=True,
                                        padding=True,
                                        truncation='only_first',
                                        return_tensors='pt',
                                        return_token_type_ids=False,
                                        return_attention_mask=True,
                                        return_overflowing_tokens=False)
```
### Step 2A: Tokenized text is put in a dictionary as follows:
```
        batch = {
            'user_input_pretok': batch_user_input_pretok,
            'nnIn_ids': batch_nnIn_ids,
            'ids': batch_ids,
            'labels': batch_token_labels
        }
        return batch
```

### Step 2B: Using PDB, I check that the tokenized text is of type BatchEncoding and that the tokenizer is Fast:
```
(Pdb) type(batch['nnIn_ids']), batch['nnIn_ids'].is_fast, self.tokenizer.is_fast
(<class 'transformers.tokenization_utils_base.BatchEncoding'>, True, True)
Note that the tokenizer is Fast, batch['nnIn_ids'] is Fast and it is of type BatchEncoding
```
### Step 3: Control goes to Lightning, and then to the predict function where I use PDB to make the same checks as follows:
```
(Pdb) type(batch['nnIn_ids']), batch['nnIn_ids'].is_fast, self.tokenizer.is_fast
(<class 'transformers.tokenization_utils_base.BatchEncoding'>, False, True)
```
Why did batch['nnIn_ids'] switched to non-Fast from Fast?
Note that the tokenizer is Fast but batch['nnIn_ids'] is Non-Fast even though it is of type BatchEncoding


### Expected behavior

A text is tokenized using a Fast tokenizer, and the tokenized-text is of type BatchEncoding and it is also Fast.
After this tokenized text is passed to another function through Lightning, it becomes non-Fast even though the tokenized-text remains of type BatchEncoding and the tokenizer remain Fast. The expected behavior of the tokenized-text should remain Fast.

cc @ArthurZucker <|||||>Hey, could you provide a full reproduction script? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.