@jmaronas: It looks good. Do you think it will behaivor as nn.tanh() layer? The purpose of the normalization layer is similar tanh layer, just normalize the input to zero mean and unit variance instead of [-1,1]