Hmrishav_Bandyopadhy:

What am i missing here?


So the thing we discussed above is that while the default init is expressed as kaiming init times some gain factor, it is not kaiming init, as the gain factor is “bogus” w.r.t. kaiming init and the activation function and only serves to reproduce ancient behaviour.
Happily, @Kushaj started work on it.