Solved by MariosOreo in post #2 
              
              
                Hello, 
The model parameters are registered in optimizer.param_groups[0]['params']. So I think you could create two separate dict for PReLU and others. It may not be the best solution, but it works simply. 
prelu_ = []
others_ = []

for module in model.children():
    if isinstance(module, nn.PReLU)…