If your original model uses nn.ReLU modules for each of these activation functions (i.e. not the functional API via F.relu not reusing the same nn.ReLU module), you could replace each nn.ReLU with an nn.Sequential container containing the nn.ReLU and your SimpleCustomLayer.