WER = 100% !! (Whisper medium)
Hello everyone,

I am having an issue when finetuning OpenAI's Whisper Medium on Mozilla's Common Voice 11 Dataset with the Arabic language.
The training and validation loss are both decreasing but the WER is being 100% after some steps (specially when the loss becomes < 1) and I see that the model is performing well and that WER is just miscalculated.
![hf_issue](https://user-images.githubusercontent.com/96656595/232808398-936c0e2b-f8b3-4289-a6bd-f21f64236efe.png)

Notes :

- This error is just happening with the medium model, other models (small, tiny, large-v2 ,etc.) are working fine.
- I am following the famous blog about Whisper's finetuning (https://huggingface.co/blog/fine-tune-whisper).
@sanchit-gandhi 
Hi @Seif-aber, thanks for raising this issue! 

So that we can best help, could you share the running environment: run `transformers-cli env` in the terminal and copy-paste the output. 

Have you looked any of the inputs to/ outputs of the model when this occurs? After the model has finished training, if you feed a single sample to the model to predict in eval model, what does the prediction look like? 
<|||||>Hey @Seif-aber! I believe I answered a duplicate of your question on the Hugging Face Hub earlier today: https://huggingface.co/spaces/openai/whisper/discussions/84#64466139e113660053727da7

My suggestions were similar to those of @amyeroberts - let's take a look at the predictions the model is making to work out what's going on.<|||||>Addressed on the HF Hub: https://huggingface.co/spaces/openai/whisper/discussions/84#644aa699af97dfd24c0e0767