I modified my code to
    N, T, D = x.size()
    _, H = h0.size()
    h = torch.zeros(N, T, H, dtype=torch.double, device='cuda')
    cache = []
    tmp, c = rnn_step_forward(x[:, 0, :], h0, Wx, Wh, b)
    cache.append(c)
    h[:, 0, :] = tmp
    for i in range(T - 1):
      tmp, c = rnn_step_forward(x[:, i+1, :], tmp, Wx, Wh, b)
      h[:, i+1, :] = tmp
      cache.append(c)

and it works, but I still don’t know what’s the difference and what’s going wrong : (