I cannot reproduce the issue using:
model = RNN(1, 1)
x = torch.randn(1, 1)
target = torch.randint(0, 2, (1, 1)).float()
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

for epoch in range(10):
    optimizer.zero_grad()
    prev = torch.randn(1, 64)
    output, h = model(x, prev)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    prev = h.detach()

Could you post an executable code snippet, which would reproduce the error you are seeing?