Hi,
The torch.no_grad() hides these ops from the autograd engine so that when you call .backward(), they won’t be taken into account. So this is the right use here.
The error you’re seeing is different: when the forward pass needs some Tensor to be able to compute the backward pass, this Tensor is saved. But if the user overrides the content of that Tensor we saved, then we can’t compute the backward pass anymore. (the fact that you override it in a differentiable manner or not does not matter here. The value that we need does not exist anymore).
I think the issue here is that you’re missing a detach() when you do hLast = hNext.
Unfortunately, because there is no operation done here, this does not work nicely with the torch.no_grad() 
In python, doing this only assigns the same python object to a new name. So no pytorch op is called and we cannot detect this automatically…
You will need to explicitly .detach() here or do any op like hLast = hNext.view_as(hNext) or hLast = hNext.clone() .