Hi,
As i understand from this, linear layers are initialized by kaiming initialization. This initialization is supposed to give output tensors of mean 0 and std 1.
However, I noticed some contradicting stuff–
>>> k=nn.Linear(512,512)
>>> k1=torch.rand(512)
>>> out=k(k1)
>>> print(out.std(),out.mean())
tensor(0.3505, grad_fn=<StdBackward0>) tensor(0.0181, grad_fn=<MeanBackward0>)

Now, when i force kaiming initialization on the layer by –
 >>> nn.init.kaiming_uniform_(k.weight,mode='fan_in')
Parameter containing:
tensor([[-0.0134,  0.0247, -0.0817,  ..., -0.0807, -0.0330, -0.0881],
        [-0.0641,  0.0447,  0.0645,  ...,  0.1048, -0.0307,  0.0989],
        [ 0.0745, -0.0076,  0.0161,  ...,  0.0252,  0.0285, -0.0527],
        ...,
        [ 0.0190, -0.0529, -0.0549,  ..., -0.0369,  0.0331,  0.0136],
        [ 0.0347,  0.0516,  0.0108,  ..., -0.0772,  0.0027,  0.0584],
        [-0.0236,  0.0565,  0.0082,  ...,  0.0717, -0.0619, -0.0772]],
       requires_grad=True)
>>> out=k(k1)
>>> print(out.std(),out.mean())
tensor(0.8420, grad_fn=<StdBackward0>) tensor(-0.0287, grad_fn=<MeanBackward0>)

I get the tensors nearing the actual mean and std values. What am i missing here?
Thanks