Solved by ptrblck in post #2 
              
              
                I think the set_to_none attribute is set to False by default for backward compatibility reasons. 
Note that it’s a performance improvement (avoiding an accumulation kernel to add the new gradients to zeroes) and wasn’t introduces to dodge the side effect of parameter updates with a zero gradient for…