Thanks for the quick reply!
My batch size is already 1, so no luck there. torch.utils.checkpoint is interesting! My model consists of transformers and classification heads on top. Transformers are already using gradient_checkpointing, but I’ll try to checkpoint the classification heads now!
Thanks for the tip