Solved by Krish in post #2 
              
              
                The kernels used by default in Pytorch use the He initialisation from this paper: 
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification