Solved by Varal7 in post #2 
              
              
                Hi, yes distributed uses grad() internally. 
There are plans to make checkpointing compatible with distribute here: New utils.checkpoint rollout. · Issue #65537 · pytorch/pytorch · GitHub