Solved by tom in post #4 
              
              
                I think you need to detach both hiddens because the hiddens that are output from the LSTM will require grad. 
I would recommend doing so when you actually store them rather than before (but probably with .detach() rather than the inplace variant). 
Best regards 
Thomas