uapatira:

In PyTorch, l1/l2 weight decay is defined on the optimizer directly. If I wanted to create a new activation layer where my parameters were learned but were not regularized, how can I go about doing that considering nn.Parameter only exposes requires_grad?


As weight decay is defined on the param_group level, you could pass the αs in a separate param group and specify weight_decay=0 for them. (or use a separate optimizer).
Alternatively - and I don’t think the overhead of taking is that terrible, you could manually add the weight decay to your loss and not use the optimizer-provided version.
Best regards
Thomas