Solved by Usama_Hasan in post #4 
              
              
                I don’t think that is the case, as I mentioned above it calculates the standard deviation of the weights considering the next non-linearity which has to be applied, It doesn’t apply relu to them. 
You should check the implementation in pytorch, I have shared the link above. 
Take this example, 
w =…