IndexError running ESMFold
### System Info

CentOS 7, transformers-4.25.0.dev0, Python 3.10.6

### Who can help?

@Rocketknight1 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running the following example script (added in #20000) is throwing an IndexError for me.

I tried both the `Rocketknight1/esmfold_v1` and `facebook/esmfold_v1` model repositories

```python
from transformers import AutoTokenizer, EsmForProteinFolding

model = EsmForProteinFolding.from_pretrained("facebook/esmfold_v1")
tokenizer = AutoTokenizer.from_pretrained("facebook/esmfold_v1")
inputs = tokenizer(["MLKNVQVQLV"], return_tensors="pt")  # A tiny random peptide
outputs = model(**inputs)
folded_positions = outputs.positions
```

```
Traceback (most recent call last):
  File "test_esmfold.py", line 7, in <module>
    outputs = model(**inputs)
  File "torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "transformers/models/esm/modeling_esmfold.py", line 2121, in forward
    esmaa = self.af2_idx_to_esm_idx(aa, attention_mask)
  File "transformers/models/esm/modeling_esmfold.py", line 2211, in af2_idx_to_esm_idx
    return self.af2_to_esm[aa]
IndexError: index 24 is out of bounds for dimension 0 with size 22
```

### Expected behavior

No IndexError
I guess you need to call
```
inputs = tokenizer(["MLKNVQVQLV"], return_tensors="pt", add_special_tokens=False)
```
code example was already updated:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/esm/modeling_esmfold.py#L2103
<|||||>Thanks @maxjeblick! And yes - the `ESMFold` tokenizer doesn't use special tokens, but the other `ESM` tokenizers do. I'll see if I can set this in the config so that users don't have to keep remembering it, because I kept getting errors from forgetting it too!