When you run a rnn net over a sequence of data, the output has two tensors: (output, h_n). While h_n is the hidden state from the last timestamp, the tensor output actually has the hidden states for every timestamps in the sequence.
rnn_net = nn.RNN(4, 3)
X = torch.rand((6,5,4))
out, hidden = rnn_net(X)
out.shape
hidden.shape

You will see the shape of out is torch.Size([6, 5, 3]) which represents (seq_len, batch, hidden_size)