Solved by vdw in post #2 
              
              
                The problem that I currently see is that you use the output of the self.lstm for the next layer. Let me use different names to avoid confusion: 
 output, (h_n,c_n) = self.lstm(t, (self.hidden_cell(t)))

Now output should has the shape (20, 1, 32); note that output gives you the hidden states for all…