Solved by tom in post #3 
              
              
                Just like for the lr in the snippet below the note you cite, you pass weight_decay globally set to what you need and add a weight_decay : 0.0 entry to the dictionary for the param group with prelu parameters. 
model = torch.nn.Sequential(torch.nn.Linear(3, 4), torch.nn.PReLU())

prelus = {n for n,…