I think you are wrong as I can’t think of a limitation why this wouldn’t work or are you seeing any issues while using both utilities?



 ben9004:

I think this should not be together because Gradient Checkpointing doesn’t utilize some of it’s layer’s computational graph and also off their requires_grad flag


That’s not entirely true. Gradient checkpointing will recompute specific operations to trade compute for memory. However, the result (compared to a non checkpointed run) should be the same.