Solved by albanD in post #2 
              
              
                Hi, 
Depending on the particular model and training loop, it may improve perf and not. 
Note that a simpler way to do this is via the regular zero grad: model.zero_grad(set_to_none=True).