@jpeg729 Thank you for your answer. I am adding this comment which can further help and elaborate on the statement "Am I right in saying that your training loop doesn’t detach or repackage the hidden state in between batches?"
Detach is very well explained but repackage is what caught my eyes and wanted an explanation. I guess I figured that out and which in my understanding is as follows:
>>> a = torch.tensor([1,2,3.], requires_grad = True)
>>> out = a.sigmoid()
>>> out.sum().backward()
>>> a.grad # This will output following
tensor([0.1966, 0.1050, 0.0452])

But when we run the following code again :
>>> out.sum().backward() # when we run this again then following runtime error is in output:
[RuntimeError]: Trying to backward through the graph a second time, 
but the buffers have already been freed. 
Specify retain_graph=True when calling backward the first time.

This error because the graph has been cleared as indicated and hence can not backpropagate. Thus, if we want to backpropagate again then we must use <retrain_graph=True> or create the graph again (which is like calling the forward  function of the model to create the graph again) by using the following:
>>> out = a.sigmoid()
>>> out.sum().backward()
>>> a.grad # This will output following. One extra thing to notice here is 
# that gradient has been added over the last one as we did not perform
# < a.grad.data.zero_()> to reset the gradient thus is accumulated. 
tensor([0.3932, 0.2100, 0.0904])

I hope this further explains and make thing more clear.
Please comment on this if something is not fine.