Move the self.flatten_parameters() call in def forward to def __init__. It has no effect on CPU, but on CUDA it will modify the LSTM’s parameters part way through the computation which would break backwards (leading to the exception).