In fact, the out is the hidden state from self.lstm, we use this hidden state to make a prediction by forwarding it into the Linear layer. Even you only has one label, then the Linear layer should be nn.Linear(hidden_size, 1). On the other hands, when you extract out[:, :, -1], you only return one value of a hidden state. It may be not enough to learn anything.