I’m pretty new to PyTorch and I’m working on a network which uses some PReLU activation layers and an Adam optimization. Since I want to try Adam with weight decay, I’ve been reading that I should not use it with PReLU because its “a” parameter would converge to some small value after some epochs and I would essentially end up with a ReLU or a LeakyReLU.
To adress this issue, I was thinking about excluding the PReLU layers from Adam’s params. The problem is that I don’t know how to do it and I think, in this case, I would need another Adam optimizer set with weight decay = 0 dedicated only to my PReLU layers, as it would not be optimized through epochs and its “a” parameter would not get updated.
Is this the right way to deal with this issue? If so, how could I exclude particular layers from Adam and add them to another Adam optimizer set with weight decay = 0? Thanks in advance.