Thanks for the clarification.
I guess you are just changing the model without modifying the forward pass.
The error might be thrown, if you forget to pass a state tensor to the LSTM.
Have a look at the docs 57.
Since hidden and state are assumed to be passed as a tuple, hidden will be sliced in case you pass it without state, which yields this size mismatch error.