Does not using clone() produces an error? If so, I suggest removing the clone and instead change the transpose_ to the non in-place version and see if that works.
I guess that the transpose_ would still be needed if batch_first of the lstm is set to true. Depending on ur case this might be wrong.
If u don’t need the hidden or the cell states(the cell states are likely to be useless in most cases), there is no need to keep the variables around.