I am unable to wrap my head around how the back propagation will work after doing this. I mean the gradient w.r.t. parameters in FC layer1 will depend on out value. But out value has changed.
Basically, W (a parameter in FC layer 1) has gradient = dLoss/dOut * dOut/dW. Now, which out value will be used for this? out or out = out*out ?