My understanding is that if we don’t detach it, then norm will be a variable and PyTorch will aim at optimizing its values in the backward phase when calculating the gradients. But we don’t really want to optimize the norm values.