Did you wind up solving this? I’m hoping to do the same thing to an existing, pre-trained network (VGG11), but I’m not sure I follow @apaszke’s suggested approach.
It sounds like I’ll need to implement a new mixed layer to replace an existing ReLU layer. In my new mixed layer I’ll need to generate a set of masks for each activation function I intend to use. Then I need to pass each masked input to the corresponding activation function, and assign the output of these to the corresponding masked output.
Does that all sound correct/possible?