I find that the gain value in the default initialization is sqrt(5).
(means a=sqrt(5) in the code below)
torch.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')

but based on the paper, if my activation function are all ReLU, i should set gain value equal to 0.
So i set a=0 in my code, but got NAN loss. What’s the problem?