Should I be sending h_n or lstm_output to my linear layer? Perhaps that is where I am making the mistake.  If I am to send h_n then when is lstm_output used or sent anywhere?  This is a many to many LSTM.  I am taking in 20x28x30495 and outputting 20x28x30490.