Hello,
The model parameters are registered in optimizer.param_groups[0]['params']. So I think you could create two separate dict for PReLU and others. It may not be the best solution, but it works simply.
prelu_ = []
others_ = []

for module in model.children():
    if isinstance(module, nn.PReLU):
        prelu_.append(module.a)
    else:
        others_.append(module.weight) # omit bias

# you could use one optimizer like this
optimizer = torch.optim.Adam([{'params': prelu_, 'lr': prelu_lr, 'weight_decay': 0},
                              {'params': others_, 'lr': lr, 'weight_decay':wd}])

see more details here. 6