Here maybe a bug, when we load staged checkpoint
https://github.com/huggingface/transformers/blob/40d98ebf50c4662bcd6dce6395bbed0b2142ea52/src/transformers/trainer.py#L458

I met this bug when I used the setting below:

global_steps = 2748
len(train_dataloader) = 27484
gradient_accumulation_steps = 4

In the original code, "steps_trained_in_current_epoch" will be 2748 !  BUT this variable should be  2748*4 = 10,992

the code I suggested is below:

```
epochs_trained = (self.global_step * self.args.gradient_accumulation_steps) // len(train_dataloader)
steps_trained_in_current_epoch = (self.global_step * self.args.gradient_accumulation_steps) % len(train_dataloader)
```
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I'm also puzzled by this. The calculations here seems incorrect.<|||||>To me these calculations are not incorrect if we take `step` as optimization steps, however `steps_trained_in_current_epoch` is wrongly used to skip training batches without considering gradient accumulation. 

+1 for the proposed calculation for `steps_trained_in_current_epoch` as the number of batches to be skipped.<|||||>@sgugger might be interested in this.<|||||>There is indeed a problem, but only with `steps_trained_in_current_epoch`. The `global_step` variable represents the number of optimization steps, not the number of batches seen. The variable `num_update_steps_per_epoch` take this into account so `epochs_trained` is correct. `steps_trained_in_current_epoch` represents the number of update steps to skip but is used as the number of batches to skip, so either need to multiply it by the `gradient_accumulation_steps` (and rename it for clarity) or skip  `gradient_accumulation_steps` batches before subtracting 1 to it later in the loop.

This also shows that we direly miss a test to check resuming training works with gradient accumulation. I can look into this when I have a bit of time, but will be fairly busy with the preparation for v4.