The problem that I currently see is that you use the output of the self.lstm for the next layer. Let me use different names to avoid confusion:
 output, (h_n,c_n) = self.lstm(t, (self.hidden_cell(t)))

Now output should has the shape (20, 1, 32); note that output gives you the hidden states for all 20 steps. The problem is that after:
t = output.view(-1, self.hidden_size)

t has the shape (20, 32). For the linear layer that means that you give it a batch of size 20. The linear layer doesn’t now or care that the 20 comes from the number of steps. It’s a semantic error. Hence your output shape is (20, 1). But what you want is (1, 1) since your batch size and output size is 1.
You have to use only the last layer of last hidden state h_n[-1] instead of output. h_n has the shape (2, 1, 32) since you have 2 layers. This h_n[-1] has the shape (1, 32). And then you also don’t need the view() no more. You can do directly
t = self.linear(h_n[-1])

Which will give you the right final output with a shape of (1, 1) or (batch_size, 1) once you have batches with more then one sequence.