In Xavier uniform weight init which you can find in eq.16 of the paper you have referenced and also PyTorch docs:
image796×497 28.2 KB
Where in the paper, the target activation function was ReLU which has gain=1. Based on this, default weight init of Linear or Conv layers in PyTorch where follows Kaiming He uniform but the gain and negative slope has been defined in a way that it is a simple uniform as the way it has been explained in the thread I posted first.
image777×494 41.8 KB
One of the differences is that in Xavier you incorporate both fan_in and fan_out in the equation but Kaiming only uses one. Note that you can obtain Xavier from Kaiming by setting negative_slope=0 and fan_out=0. But it does not work as fan_out=0 does not make sense in neural networks.
In the same way that you can get Xavier from Kaiming, you can get standard uniform from Kaiming too.