In addition to Peter’s spot-on comments about symmetry breaking, there is a the lottery ticket hypothesis 4, roughly speaking the theory that (overparametrised by traditional standards) NNs are “looking in many places of the parameter landscape, thereby picking up some useful ones”.
Weight initialization in particular is something  that has been identified as fairly important and I can recommend spending thought on it - PyTorch inherits the initializations mostly from Torch, and might not always reflect the latest advice of how to do it. Most stock modules have a method reset_parameters that has the default (e.g. do ?? torch.nn.Conv2d.reset_parameters to see the source in IPython/Jupyter).
In contrast to weight, bias can, in my experience, often just be zeroed.
Best regards
Thomas