[ASR example] Can't reproduce the same WER with the same seed
## Environment info
- `transformers` version: 4.12.0.dev0
- Platform: Linux
- Python version: 3.8.10 
- PyTorch version (GPU?): 1.7.1+cu110
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @anton-l

## Information
I was trying the ASR example: https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition
However, I found that even after setting the seed, I got two different WERs in the end. I'm not sure how I can make it reproducible?

Some more information: I used common voice 7.0 ug data, here are two testing results I got with the same seed=42:
{
    "test_loss": 0.4873116910457611,
    "test_runtime": 165.4426,
    "test_samples": 2620,
    "test_samples_per_second": 15.836,
    "test_steps_per_second": 1.983,
    "test_wer": 0.46173210049862035
}
{
    "test_loss": 0.47727492451667786,
    "test_runtime": 166.5077,
    "test_samples": 2620,
    "test_samples_per_second": 15.735,
    "test_steps_per_second": 1.97,
    "test_wer": 0.46052185699762793
}
Though they are very close, they are not exactly the same. When I run the code on smaller datasets, the variance is larger.

Thanks!
Hey @ZhangShiyue, 

Thanks for the report! Hmm, we do call the function `set_seed(...)` [here](https://github.com/huggingface/transformers/blob/95bab53868a91b4809bd5281a72b5f326853e31f/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L332) which should make everything reproducible...do you train on single GPU? <|||||>yes, I noticed it. So it also surprised me when I got different results with the same seed. 
Yes I train on a single GPU
thanks!<|||||>I will take a look to see whether I can make it fully reproducible on a tiny training run :-) I'll keep you posted!<|||||>Thank you!<|||||>Hey @ZhangShiyue, 

sorry to answer only now. It's quite difficult to debug such reproducibility problems. I've created a dummy script and ran it a couple of times and it always gave me the same result. The script is:

```python
CUDA_VISIBLE_DEVICES="0" python run_speech_recognition_ctc.py \
        --dataset_name="common_voice" \
        --model_name_or_path="ntu-spml/distilhubert" \
        --dataset_config_name="ab" \
        --output_dir="./dummy" \
        --overwrite_output_dir \
        --num_train_epochs="3" \
        --per_device_train_batch_size="4" \
        --gradient_accumulation_steps="1" \
        --learning_rate="5e-5" \
        --warmup_steps="500" \
        --evaluation_strategy="steps" \
        --text_column_name="sentence" \
        --save_steps="500" \
        --eval_steps="500" \
        --logging_steps="1" \
        --layerdrop="0.0" \
        --save_total_limit="1" \
        --mask_time_prob="0.3" \
        --mask_time_length="10" \
        --mask_feature_prob="0.1" \
        --mask_feature_length="64" \
        --freeze_feature_extractor \
        --chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \
        --fp16 \
        --group_by_length \
        --do_train --do_eval \
        --gradient_checkpointing \
```

and I've gotten the following result three times in a row:
```
{'loss': 33.9851, 'learning_rate': 3.0000000000000004e-07, 'epoch': 1.17}                     
{'loss': 28.627, 'learning_rate': 4.0000000000000003e-07, 'epoch': 1.33}                      
{'loss': 46.3949, 'learning_rate': 5.000000000000001e-07, 'epoch': 1.5}                       
{'loss': 26.7014, 'learning_rate': 6.000000000000001e-07, 'epoch': 1.67}                      
{'loss': 21.1327, 'learning_rate': 7.000000000000001e-07, 'epoch': 1.83}                      
{'loss': 26.1967, 'learning_rate': 8.000000000000001e-07, 'epoch': 2.0}                       
{'loss': 37.1694, 'learning_rate': 9e-07, 'epoch': 2.17}                                      
{'loss': 30.2389, 'learning_rate': 1.0000000000000002e-06, 'epoch': 2.33}                     
{'loss': 30.4334, 'learning_rate': 1.1e-06, 'epoch': 2.5}                                     
{'loss': 31.5205, 'learning_rate': 1.2000000000000002e-06, 'epoch': 2.67}                     
{'loss': 28.4714, 'learning_rate': 1.3e-06, 'epoch': 2.83}                                    
{'loss': 21.3841, 'learning_rate': 1.4000000000000001e-06, 'epoch': 3.0}                      
100%|█████████████████████████████████████████████████████████| 18/18 [00:03<00:00,  6.20it/s]

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 3.6622, 'train_samples_per_second': 18.022, 'train_steps_per_second': 4.915, 'train_loss': 29.691171010335285, 'epoch': 3.0}
100%|█████████████████████████████████████████████████████████| 18/18 [00:03<00:00,  4.92it/s]
Saving model checkpoint to ./dummy
Configuration saved in ./dummy/config.json
Model weights saved in ./dummy/pytorch_model.bin
Configuration saved in ./dummy/preprocessor_config.json
***** train metrics *****
  epoch                    =        3.0
  train_loss               =    29.6912
  train_runtime            = 0:00:03.66
  train_samples            =         22
  train_samples_per_second =     18.022
  train_steps_per_second   =      4.915
12/10/2021 13:45:39 - INFO - __main__ - *** Evaluate ***
***** Running Evaluation *****
  Num examples = 9
  Batch size = 8
100%|███████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 61.92it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_loss               =    23.3384
  eval_runtime            = 0:00:00.26
  eval_samples            =          9
  eval_samples_per_second =     33.613
  eval_steps_per_second   =       7.47
  eval_wer                =        1.0
Dropping the following result as it does not have all the necessary fields:
{'dataset': {'name': 'COMMON_VOICE - AB', 'type': 'common_voice', 'args': 'Config: ab, Training split: train+validation, Eval split: test'}}
```

The loss values are also exactly the same. 

My environment is one GPU (`GeForce RTX 3070 Mobile / Max-Q`) and the following library settings:

```
- `transformers` version: 4.14.0.dev0
- Platform: Linux-5.15.5-76051505-generic-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.0 (True)
- Tensorflow version (GPU?): 2.7.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.3.6 (cpu)
- Jax version: 0.2.25
- JaxLib version: 0.1.73
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```<|||||>Could you maybe try to run the same script a couple of times from your side (it's very fast - just 20 seconds) to see if you can at least reproduce the results of this script? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.