Note: you need torch>=1.7 to enjoy this feature.
You can also see a variety of benchmarks on tf32 vs other precisions:
RTX-3090 and
A100.
Gradient Accumulation
Since gradient accumulation essentially is identical to having a larger batch size, just as with the larger batch size here you are likely to see a 20-30% speedup due to the optimizer running less often. For example, see benchmarks for RTX-3090
and A100.
To activate this feature in 🤗 Trainer add --gradient_accumulation_steps 4 to its arguments (experiment with the value to get the best performance).
It's important to remember that using gradient accumulation you may end up with a much larger effective batch size, so you may need to adjust the learning rate, its warm up and for very short datasets it'll impact the loss as the training will end up doing less steps than normal.
