What is the different performance ranges, with and without initializations?
What type of models are you using? Generative models, say GAN’s, are very sensitive to initializations.
Empirically, I have observed that the using Batch Norm Layers tend to ease the dependency on a good initialization.
Xavier’s initialization can be used to prevent the exploding and vanishing gradients problem.