Well, as can be seen here 3, the variable lstm_output is of size torch.Size([20, 28, hidden_size]), i.e. you get the hidden activations of every step of the sequence. So, when you flatten the variable, you should get a tensor of size torch.Size([20, 28*hidden_size]). Judging by the dimensions of the output layer in your runtime error (in_features==512), 28*512 == 14336, which is what you are getting when you flatten the variable.