SimonW:

Being normally distributed doesn’t mean that it has small stddev. Being standard normally distributed will only have small mean.


True. Thx for the advice.
But from a deeper look, I found out that I got nan only when the hidden unite are all 0. That means that both mean and std are 0. For some reason, if you try this:
import torch

loss_fn = torch.nn.MSELoss()
x = torch.autograd.Variable(torch.zeros(1, 5), requires_grad=True)
mean = x.mean(-1, keepdim=True)
std = x.std(-1, keepdim=True)
r = (x - mean)/(std + 1e-6)
loss = loss_fn(r, torch.autograd.Variable(torch.ones(1, 5)))
loss.backward()
r.grad

You got a None grad, which seems odd. Any intuition why this is happening ?