Thx! I will check it later.
And I noticed you once said in another topic that torch.utils.checkpoint may help when Cuda out of memory with batch size equals to 1. Does that fit to my situation? Appreciate so much !