Hello 乃仁 梁!



 111296:

there is a network saying that the weight decay specified by the optimizer weight_decay parameter of torch.optim is for all parameters in the network


If you wish to turn off weight decay for your network biases, you may
use “parameter groups” to use different optimizer hyperparameters to
optimize different sets of network parameters.
Here’s a brief example:
import torch
lin = torch.nn.Linear (3, 1)
opt = torch.optim.SGD ([
    {'params': lin.weight, 'lr': 0.1, 'weight_decay': 0.5},
    {'params': lin.bias,   'lr': 0.1, 'weight_decay': 0.0}
])

opt will use a learning rate of 0.1 for all of lin's parameters – both
weight and bias – but will only use a weight decay of 0.5 for weight
and no weight decay (weight_decay = 0.0) for bias.
Best.
K. Frank