In my case, I wanted the weights to be updated after hitting them with noise. Yet, you can use torch.no_grad() to prevent the gradient from updating your weights. You can also use detach() method, which constructs a new view on a tensor which is declared not to need gradients , i.e., it is to be excluded from further tracking of operations, and therefore the subgraph involving this view will not be recorded.