Solved by ptrblck in post #2 
              
              
                Not necessarily. The regularization term will modify all parameters which are is its computation graph. 
Have a look at this dummy example: 
model = nn.Sequential(
    nn.Linear(1, 1, bias=False),
    nn.Sigmoid(),
    nn.Linear(1, 1, bias=False)
)

criterion = nn.MSELoss()
x = torch.randn(1, 1)
tar…