I wouldn’t call it a bizarre behavior and would expect that the loss might get stuck using a wrong initialization method.
In my opinion, sane weight init functions were a key to be able to train deep neural networks at all.