abandon_tf:

Every loop in train I want to save self.num_steps   hidden state for further use.


This is the problem. Every variable you save has the computation graph associated. If you use that in a future forward, then the previous computation graph needs to be there (retain_graph=True) or it will complain since it can’t properly backprop.
Why do you need those? Could you just backprop a limit number of steps?