RuntimeError: unscale_() has already been called on this optimizer since the last update()
### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: Yes, T4
- Using distributed or parallel set-up in script?: No

### Who can help?

@younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:

1. Run https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zbf2-9VQTtGJ24k?usp=sharing

### Expected behavior

Last cell in the notebook should run the training. Instead, it's showing the error message

`RuntimeError: unscale_() has already been called on this optimizer since the last update().`
The notebook was working on the 25th May
Looking at the traceback:
```bash
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 23>:23                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1661 in train                    │
│                                                                                                  │
│   1658 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1659 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1660 │   │   )                                                                                 │
│ ❱ 1661 │   │   return inner_training_loop(                                                       │
│   1662 │   │   │   args=args,                                                                    │
│   1663 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1664 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:2008 in _inner_training_loop     │
│                                                                                                  │
│   2005 │   │   │   │   │   │   │   │   args.max_grad_norm,                                       │
│   2006 │   │   │   │   │   │   │   )                                                             │
│   2007 │   │   │   │   │   │   else:                                                             │
│ ❱ 2008 │   │   │   │   │   │   │   self.accelerator.clip_grad_norm_(                             │
│   2009 │   │   │   │   │   │   │   │   model.parameters(),                                       │
│   2010 │   │   │   │   │   │   │   │   args.max_grad_norm,                                       │
│   2011 │   │   │   │   │   │   │   )                                                             │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:1878 in clip_grad_norm_        │
│                                                                                                  │
│   1875 │   │   │   # `accelerator.backward(loss)` is doing that automatically. Therefore, its i  │
│   1876 │   │   │   # We cannot return the gradient norm because DeepSpeed does it.               │
│   1877 │   │   │   return None                                                                   │
│ ❱ 1878 │   │   self.unscale_gradients()                                                          │
│   1879 │   │   return torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=norm_type)  │
│   1880 │                                                                                         │
│   1881 │   def clip_grad_value_(self, parameters, clip_value):                                   │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:1841 in unscale_gradients      │
│                                                                                                  │
│   1838 │   │   │   for opt in optimizer:                                                         │
│   1839 │   │   │   │   while isinstance(opt, AcceleratedOptimizer):                              │
│   1840 │   │   │   │   │   opt = opt.optimizer                                                   │
│ ❱ 1841 │   │   │   │   self.scaler.unscale_(opt)                                                 │
│   1842 │                                                                                         │
│   1843 │   def clip_grad_norm_(self, parameters, max_norm, norm_type=2):                         │
│   1844 │   │   """                                                                               │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/cuda/amp/grad_scaler.py:275 in unscale_            │
│                                                                                                  │
│   272 │   │   optimizer_state = self._per_optimizer_states[id(optimizer)]                        │
│   273 │   │                                                                                      │
│   274 │   │   if optimizer_state["stage"] is OptState.UNSCALED:                                  │
│ ❱ 275 │   │   │   raise RuntimeError("unscale_() has already been called on this optimizer sin   │
│   276 │   │   elif optimizer_state["stage"] is OptState.STEPPED:                                 │
│   277 │   │   │   raise RuntimeError("unscale_() is being called after step().") 
```

I think one of the recent PR of accelerate + trainer has touched something by mistake. My gut feeling is that maybe it needs a proper gradient accumulation support.
I can have a look at it, I am curious if @pacman100 or @sgugger have any thoughts also on that<|||||>Also cc @muellerzr <|||||>Looking into this
<|||||>Have the same issue, watching<|||||>Yes, narrowing down on it, will be raising a PR in 10 minutes
<|||||>@tehistarkderek in the meantime, if you need to do something now you can replace the `pip install git+https:...` for transformers with:
```python
pip install git+https://github.com/huggingface/transformers@de9255de27abfcae4a1f816b904915f0b1e23cd9
```<|||||>Hello @younesbelkada, can you confirm if the above PR fixes the issue? I've checked it on my end and it is working as expected <|||||>It works like charm @pacman100 , thanks so much for taking care of this!<|||||>@younesbelkada I have reinstalled from source but I still get the error. I'm in the latest commit `transformers @ git+https://github.com/huggingface/transformers.git@fabe17a726bbf6081cfbcc975d8ac451a81f3e2d` and you can tell from the stacktrace that the line numbers are different (due to the changes merged here https://github.com/huggingface/transformers/pull/23914/files)

```╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 17>:17                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1661 in train                    │
│                                                                                                  │
│   1658 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1659 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1660 │   │   )                                                                                 │
│ ❱ 1661 │   │   return inner_training_loop(                                                       │
│   1662 │   │   │   args=args,                                                                    │
│   1663 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1664 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1995 in _inner_training_loop     │
│                                                                                                  │
│   1992 │   │   │   │   │   │   │   │   args.max_grad_norm,                                       │
│   1993 │   │   │   │   │   │   │   )                                                             │
│   1994 │   │   │   │   │   │   else:                                                             │
│ ❱ 1995 │   │   │   │   │   │   │   self.accelerator.clip_grad_norm_(                             │
│   1996 │   │   │   │   │   │   │   │   model.parameters(),                                       │
│   1997 │   │   │   │   │   │   │   │   args.max_grad_norm,                                       │
│   1998 │   │   │   │   │   │   │   )                                                             │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:1817 in clip_grad_norm_        │
│                                                                                                  │
│   1814 │   │   │   # `accelerator.backward(loss)` is doing that automatically. Therefore, its i  │
│   1815 │   │   │   # We cannot return the gradient norm because DeepSpeed does it.               │
│   1816 │   │   │   return None                                                                   │
│ ❱ 1817 │   │   self.unscale_gradients()                                                          │
│   1818 │   │   return torch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=norm_type)  │
│   1819 │                                                                                         │
│   1820 │   def clip_grad_value_(self, parameters, clip_value):                                   │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:1780 in unscale_gradients      │
│                                                                                                  │
│   1777 │   │   │   for opt in optimizer:                                                         │
│   1778 │   │   │   │   while isinstance(opt, AcceleratedOptimizer):                              │
│   1779 │   │   │   │   │   opt = opt.optimizer                                                   │
│ ❱ 1780 │   │   │   │   self.scaler.unscale_(opt)                                                 │
│   1781 │                                                                                         │
│   1782 │   def clip_grad_norm_(self, parameters, max_norm, norm_type=2):                         │
│   1783 │   │   """                                                                               │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/cuda/amp/grad_scaler.py:275 in unscale_            │
│                                                                                                  │
│   272 │   │   optimizer_state = self._per_optimizer_states[id(optimizer)]                        │
│   273 │   │                                                                                      │
│   274 │   │   if optimizer_state["stage"] is OptState.UNSCALED:                                  │
│ ❱ 275 │   │   │   raise RuntimeError("unscale_() has already been called on this optimizer sin   │
│   276 │   │   elif optimizer_state["stage"] is OptState.STEPPED:                                 │
│   277 │   │   │   raise RuntimeError("unscale_() is being called after step().")                 │
│   278                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: unscale_() has already been called on this optimizer since the last update().
```

am I missing something? I'm just running this colab demo from a live event yesterday https://colab.research.google.com/drive/1ARmlaZZaKyAg6HTi57psFLPeh0hDRcPX?usp=sharing#scrollTo=Duak7T_B3VpJ<|||||>Hi @kafkasl 
Please check: https://github.com/huggingface/transformers/issues/23935#issuecomment-1571989596