The type of norm is torch Variable. PyTorch will only calculate the the gradient of loss w.r.t to the leaf node. Since norm is not a leaf node, I do think it will be updated when we do optimizer.step(). Only emb.weight will be updated since it is of type torch.nn.Parameter and it is the learnable parameter of the module.