Evaluation of wav2vec2 model all labeled string return "<unk>" value
### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Both have same issue

- $ pip freeze |grep datasets
datasets==2.4.0

### Who can help?

@patrickvonplaten
@anton-l
@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the issue:
1. Download the [issue_report](https://drive.google.com/drive/folders/1M5xE4L_HBxBQynWyl6f1c-tJtat027d1?usp=sharing) folder to your local
2. open a command prompt and cd to the issue_report
3. run eval cmd: python ctc_finetune.py --eval
4. The loss is 1.0086 as the value of label_str always "unk", which printed at the line # 566 of [ctc_finetune.py](https://drive.google.com/file/d/1NogO0G8-RtLGaisfcmBrXh6ESKZbWfZK/view?usp=sharing)
5. To re-generate the datase cache files, please run : python customise_dataset.py

Here is the log printed at end of evaluation as following, please see the [full_log.log](https://drive.google.com/file/d/1hIMmxfLXOEm_Sx_g3100MvR2so1sthEM/view?usp=sharing) for more details :
***** Running Evaluation *****
  Num examples = 91
  Batch size = 4
100%|███████████████████████████████████████████| 23/23 [00:03<00:00,  5.54it/s]
**pred_str[0]: THERE WERE BARRELS OF WINE IN THE SHU CELLOR
label_str[0]: <unk><unk><unk><unk><unk> <unk><unk><unk><unk> <u**nk><unk><unk><unk><unk><unk><unk> <unk><unk> <unk><unk><unk><unk> <unk><unk> <unk><unk><unk> <unk><unk><unk><unk> <unk><unk><unk><unk><unk><unk>
100%|███████████████████████████████████████████| 23/23 [00:03<00:00,  6.12it/s]
***** eval metrics *****
  eval_loss               =  4704.6416
  eval_runtime            = 0:00:06.64
  eval_samples            =         91
  eval_samples_per_second =     13.697
  eval_steps_per_second   =      3.462
  eval_wer                =     1.0086

### Expected behavior

As I use original pre-trained model : facebook/wav2vec2-large-robust-ft-libri-960h for evaluation, the only changes is my customized dataset. 
I could not figure out where is wrong with my own modified scripts which had just minor change from the official example scripts. 
So I am not sure my encountered issue whether it's my scripts issue or the finetune libs issue. 
Thanks in advance for helping me on this matter.  
Are the labels in lower case?<|||||>Maybe of interest to @sanchit-gandhi as well<|||||>> Are the labels in lower case?

Yes, it's lower case

Here is the csv file looks like:
$ head dataset.csv 
path,transcription
wav/000010001.WAV,there were barrels of wine in the huge cellar
wav/000010002.WAV,she won a car because she was the twelfth person to call the radio station
wav/000010003.WAV,as they walked back they were shocked to see a pack of stray dogs circling around the car<|||||>Ok I think that's the issue. Your vocabulary likely only contains upper case letters. The tokenizer doesn't recognise lower case letters so it uses `<unk>` instead.

Try converting your transcription column to upper case and see if that fixes it.<|||||>> Ok I think that's the issue. Your vocabulary likely only contains upper case letters. The tokenizer doesn't recognise lower case letters so it uses `<unk>` instead.
> 
> Try converting your transcription column to upper case and see if that fixes it.

Yeah, that is the root cause. After I changed it to upper case, the issue go aways:
Thank you so much for the troubleshoot. 

***** Running Evaluation *****
  Num examples = 91
  Batch size = 4
100%|███████████████████████████████████████████| 23/23 [00:03<00:00,  6.42it/s]pred_str[0]: THERE WERE BARRELS OF WINE IN THE SHU CELLOR
label_str[0]: THERE WERE BARRELS OF WINE IN THE HUGE CELLAR
100%|███████████████████████████████████████████| 23/23 [00:03<00:00,  6.68it/s]
***** eval metrics *****
  eval_loss               =   118.7373
  eval_runtime            = 0:00:05.39
  eval_samples            =         91
  eval_samples_per_second =     16.856
  eval_steps_per_second   =       4.26
  eval_wer                =     0.1228


