Very strange Training Data Loss Pattern when fitting MT5 for Summarization
Hi,

I am training a MT5-small summarization model.
On tensorboard the loss function looks like this (see below).

The orange one is with batch size 3 and gradient_accumulation_steps 2 and 10 epochs. It looks normal.

The red one is with the same parameters but gradient_accumulation_steps 3 instead of 2.

The graph looks very strange. After each epoch it makes a huge jump down. Then it seems like it goes up again
between the "jumps".

For me this somehow looks like maybe a bug with gradient_accumulation.

Can someone please check this?

PS: I am using only one GPU.

PPS: The orange line is with default learing rate of 5e-5 the red graph is with learning_rate=5e-4

![grafik](https://user-images.githubusercontent.com/229382/126282933-293d8d1d-e6cc-401a-9560-7353c1d63600.png)

Pinging @patil-suraj and @patrickvonplaten for MT5<|||||>PS: I am using only one GPU.<|||||>Hi, @PhilipMay Could please mention which script are you using for this? (Official examples/custom script, Pytrich/TF/FLax ?) <|||||>> Hi, @PhilipMay Could please mention which script are you using for this? (Official examples/custom script, Pytrich/TF/FLax ?)

Oops. Sure @patil-suraj 

I am using this: https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py<|||||>The params are:
```bash
python run_summarization.py --model_name_or_path "google/mt5-small" --do_train --do_eval   \
--source_prefix "summarize: " --output_dir ./checkpoints --per_device_train_batch_size=3 \
--per_device_eval_batch_size=3 --overwrite_output_dir --predict_with_generate --max_source_length 800 \
--max_target_length 96 --save_strategy="epoch" --warmup_ratio=0.3  --dataset_name de_en_mit \
--num_train_epochs 10 --report_to tensorboard --gradient_accumulation_steps=3  --learning_rate=5e-4
```

I modified the script so that `--dataset_name de_en_mit` loads my custom csv dataset.<|||||>PPS: The orange line is with default learing rate of 5e-5 the red graph is with learning_rate=5e-4

Sorry that I had forgotten this info. I have added it again above.<|||||>Small update until the end of the training

![grafik](https://user-images.githubusercontent.com/229382/126457759-e4a5e72e-ba5f-4524-9a6f-5509ae6bb156.png)
<|||||>@patil-suraj so what do you think about this?<|||||>@patil-suraj and news about this?<|||||>Gently pinging @sgugger on this one as well since it's based on the trainer. I'm not really sure it's a bug to be honest<|||||>I'm not sure what bug in the Trainer I should be looking at. What I see is a bumpier training with a higher learning rate, which seems completely normal.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.