Solved by mcstarioni in post #2 
              
              
                Layer Normalization i used was normalizing features along axis which had a size of 1 in the last layer, so all of the output was normalized to be 0.