Gradient Checkpointing
One way to use significantly less GPU memory is to enabled "Gradient Checkpointing" (also known as "activation checkpointing"). When enabled, a lot of memory can be freed at the cost of small decrease in the training speed due to recomputing parts of the graph during back-propagation. The slowdown will depend on the model but quite often it is around 20-30%.
This technique was first shared in the paper: Training Deep Nets with Sublinear Memory Cost. The paper will also give you the exact details on the savings, but it's in the ballpark of O(sqrt(n)), where n is the number of feed-forward layers.
To activate this feature in 🤗 Transformers for models that support it, use:
python
model.gradient_checkpointing_enable()
or add --gradient_checkpointing to the Trainer arguments.
