Hi,
I’m late to the party. But I think batch normalization is exactly what you need. You can add a BN-layer before any activation: BatchNorm2d — PyTorch 1.8.1 documentation 14
The common practice is to normalize values (zero-center and divide by emprical variance) before feeding the values to a NN. Batch normalization simply extends this idea to intermediate neurons as well.