You are right! I had a small issue, somehow when I initialize my weights according to the custom init function, the training becomes much harder, but if I do hyper-parameter tuning then the model starts to train. Your suggestion on how to set the weights to custom values works well.