Solved by Nikronic in post #2 
              
              
                Hi, 
For the first question, please see these posts: 

Clarity on default initialization in pytorch
CNN default initialization understanding

I have explained the magic number math.sqrt(5) so you can also get the idea behind the relation between non-linearity and init method. Acuatlly, default initi…