trainer gradient_accumulation_steps
Hi, correct me if I am wrong.
I think it would be better to remove this line, as it ignore the case that last step in epoch and step is **larger** than gradient_accumulation_steps, which is quite usual.

https://github.com/huggingface/transformers/blob/c016dbdbdaf79339ae6d275d4651dc9f380be055/src/transformers/trainer.py#L1336
cc @sgugger <|||||>No, this line is here for the edge case of very small datasets, where one full epoch does not produce enough gradient accumulation steps, to make sure that there is still a step in that case.<|||||>Ok this seems to be a very rare case. Further more, this line will result in “drop last” for large dataset.

------------------ Original ------------------
From: Sylvain Gugger ***@***.***&gt;
Date: Wed,Nov 10,2021 3:45 AM
To: huggingface/transformers ***@***.***&gt;
Cc: cOng ***@***.***&gt;, Author ***@***.***&gt;
Subject: Re: [huggingface/transformers] trainer gradient_accumulation_steps (Issue #14309)






No, this line is here for the edge case of very small datasets, where one full epoch does not produce enough gradient accumulation steps, to make sure that there is still a step in that case.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @sgugger 
As @congchan mentioned "drop last", it is very common for large dataset.

It will cause that , some samples may be not used for optimizer.
(When total_batched_samples=15 and gradient_accumulation_steps=4, samples with 13,14,15-th batch will not be used for optimizer) 

Is this okay? As I checked the latest trainer code, it does not updated.
(https://github.com/huggingface/transformers/blob/bacaab1629972b85664fe61ec3caa4da7b55b041/src/transformers/trainer.py#L1965)