audio classification example script RuntimeError on evaluation
### System Info

accelerate==0.21.0.dev0
OS: Ubuntu 20.04
- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.0-57-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: RTX 2080TI, also happened on A100 (colab)
- Using distributed or parallel set-up in script?: no
- The rest are from the latest hugging face pytorch docker image

### Who can help?

@sanchit-gandhi @sgugger @albertvillanova

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

` 693/6930 [08:55<1:18:59,  1.32it/s][INFO|trainer.py:3074] 2023-06-24 18:44:28,008 >> ***** Running Evaluation *****
[INFO|trainer.py:3076] 2023-06-24 18:44:28,008 >>   Num examples = 5888
[INFO|trainer.py:3079] 2023-06-24 18:44:28,008 >>   Batch size = 1
{'eval_loss': 2.8346564769744873, 'eval_accuracy': 0.2105978260869565, 'eval_runtime': 70.8074, 'eval_samples_per_second': 83.155, 'eval_steps_per_second': 83.155, 'epoch': 1.0}                          
>Saving model checkpoint to wav2vec2-base-lang-id/checkpoint-693                                                                                           
[INFO|configuration_utils.py:458] 2023-06-24 18:45:38,818 >> Configuration saved in wav2vec2-base-lang-id/checkpoint-693/config.json
[INFO|modeling_utils.py:1845] 2023-06-24 18:45:39,166 >> Model weights saved in wav2vec2-base-lang-id/checkpoint-693/pytorch_model.bin
[INFO|feature_extraction_utils.py:377] 2023-06-24 18:45:39,166 >> Feature extractor saved in wav2vec2-base-lang-id/checkpoint-693/preprocessor_config.json
Traceback (most recent call last):
  File "run_audio_classification.py", line 418, in <module>
    main()
  File "run_audio_classification.py", line 392, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/transformers/src/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/transformers/src/transformers/trainer.py", line 1850, in _inner_training_loop
    self.accelerator.clip_grad_norm_(
  File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 1913, in clip_grad_norm_
    self.unscale_gradients()
  File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 1876, in unscale_gradients
    self.scaler.unscale_(opt)
  File "/usr/local/lib/python3.8/dist-packages/torch/cuda/amp/grad_scaler.py", line 275, in unscale_
    raise RuntimeError("unscale_() has already been called on this optimizer since the last update().")
RuntimeError: unscale_() has already been called on this optimizer since the last update().
`

### Expected behavior

I'd expect the script to train the model and finish successfully.
Thank you very much :)
Hey @adamkatav - do you have a reproducible code snippet we could use to emulate this behaviour?<|||||>> Hey @adamkatav - do you have a reproducibl
e code snippet we could use to emulate this behaviour?

Sure, it's a copy-paste from [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).
I'm also attaching the output file from my terminal. [hugging_face_error.txt](https://github.com/huggingface/transformers/files/11885751/hugging_face_error.txt)

I re-ran the script after a reboot on a fresh official hugging face container with a RTX 2080 TI GPU.<|||||>Hi, @adamkatav I was not reproduce the error you mentioned in the above messages, I'm on the same versions (for the most part) as mentioned.<|||||>> Hi, @adamkatav I was not reproduce the error you mentioned in the above messages, I'm on the same versions (for the most part) as mentioned.

Thank you for the reply, might it be a docker issue? Because I got the same error on colab, both environments are in a docker container.<|||||>Yes, the issue you encountered with the RuntimeError could potentially be related to the Docker environment. If you are experiencing the same error in Colab as well, it suggests that the issue might be related to the specific Docker setup or configuration you are using. Maybe try testing it outside the Docker container. <|||||>Hey @adamkatav - I'm also not able to reproduce the error using the example script. It trains for me as expected. Just as a precautionary measure, could you try installing `accelerate` / `transformers` at the last stable PyPi versions?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.