Why is my train_samples_per_second graph growing?
I'm training T5 so I'll tag @patrickvonplaten in the off chance it's something specific and not an obvious thing I'm missing.

Why is my train_samples_per_second graph linearly growing? I didn't set an increasing batch size, so it should be constant, right?

<img width="365" alt="image" src="https://user-images.githubusercontent.com/31738272/153556974-46775320-cadb-4327-8ac0-31a3ab231026.png">


Hey @drunkinlove, 

Could you specify which training you used exactly? Do you use the official HF Trainer : https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.Trainer? <|||||>@patrickvonplaten Hi! I did indeed, here are the arguments I specified:

- output_dir="checkpoints",
- logging_dir="logs",
- logging_strategy="steps",
- logging_steps=100,
- per_device_train_batch_size=4,
- num_train_epochs=100,
- max_steps=10000,
- warmup_steps=5000,
- weight_decay=0.01,
- evaluation_strategy="steps",
- save_strategy="steps",
- eval_steps=5000,
- save_steps=5000,
- save_total_limit=100,
- fp16=False,
- gradient_accumulation_steps=1

Am I even reading this graph correctly? It's supposed to be 'amount of training samples seen per second' vs 'total seconds passed', right? It looks really suspicious, I'm considering logging the time after each step myself to make my own graph.<|||||>Hey @drunkinlove,

I looked a bit into it. This graph should actually not show a line really but just a single scalar value for a single training run. If you look into this code line: https://github.com/huggingface/transformers/blob/b87c044c79a408c0a1e7f7b046b5b4ce999c2d0e/src/transformers/trainer.py#L1546,

you can see that the `train_samples_per_second` metric is computed only once during training (after the training is finished) and the tensorboard should only show a single scalar as it is done here: https://huggingface.co/patrickvonplaten/xls-r-300m-sv-cv8/tensorboard 

So, I don't really understand how it is possible that you see a linear graph there. Do you have a link to your repo on the Hub? 

The metric `train_samples_per_second` literally just says "How many training samples were treated per second" so it's a very easy formula: `number of all training samples` / `total time of training`<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.