I think you need to detach both hiddens because the hiddens that are output from the LSTM will require grad.
I would recommend doing so when you actually store them rather than before (but probably with .detach() rather than the inplace variant).
Best regards
Thomas