I tried to reproduce this issue with a small example, but if I try to pass the initial hidden state as a DoubleTensor, I get an error:
torch.manual_seed(2809)
rnn = nn.GRU(10, 20, 2)
for _ in range(100):
    input = torch.randn(5, 3, 10)
    h0 = torch.randn(2, 3, 20).double()
    output, hn = rnn(input, h0)
    print(output.abs().max(), hn.abs().max())

> RuntimeError: expected scalar type Float but found Double

Could you share a code snippet so that we could have a look?