Solved by ptrblck in post #4 
              
              
                It’s not generally true, I think, and might depend on your application. 
You can find literature about how standardizing whitens the input data and this creates a “round” loss surface, but I’m not sure if these simple abstractions are applicable for deep neural networks. 
Have a look at this post wr…