Bingchen_Liu:

However, rows 3,4,5 are different from 1,2, which is not expected.


I think that might be expected.
In 1 and 2 you are applying the weight init on the CPU, while 3 and 5 are not using the weight init at all, so these results should differ. (Also 3 and 5 seem to be the same use case, no?)
In 4 you are applying the weight init on the GPU, which will use another pseudorandom number generator as seen here:

torch.manual_seed(2809)
print(torch.randn(1))
> tensor([-2.0748])

torch.manual_seed(2809)
print(torch.randn(1))
> tensor([-2.0748])

torch.manual_seed(2809)
print(torch.randn(1, device='cuda'))
> tensor([0.5603], device='cuda:0')

torch.manual_seed(2809)
print(torch.randn(1, device='cuda'))
> tensor([0.5603], device='cuda:0')