That was very clear and to the point. Thanks a lot.
One final question, is there a way I could hook somehow and get the output of the batchnorm layer and not the relu output ? I cannot use conv layer’s output because my loss function involves a margin  where I can set the margin to one std. dev if only I get output of batchnorm layer.
Thanks again.