How to retrieve hidden states for all time steps in LSTM or BiLSTM?
According to the docs of nn.LSTM outputs: <SCODE>output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence.
h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
c_n (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len
<ECODE> If I want to get the hidden states for all t which means t =1, 2, …, seq_len, How can I do that? One approach is looping through an LSTM cell for all the words of a sentence and get the hidden state, cell state and output. I am doing a language modeling task using LSTM where I need the hidden state representations of all the words of a sentence. Any help would be appreciated!
wasiahmad
To get individual hidden states, you have to indeed loop over for each individual timestep and collect the hidden states.
smth
Isn’t output containing all the hidden states (at least h)?
ruotianluo
Thanks smth, I was also thinking about that. Just one more thing to confirm, in case if I loop over the individual timestep, will it be inefficient in terms of time complexity and will it change the loss computation for the entire network? Though i think it will not.
wasiahmad
No, the output only contains output of the last time step.
wasiahmad
It won’t change loss computation, but it will likely be much slower.
ngimel
no, output contains the hidden states for each time step, but only for the last layer in a stacked model, or the only layer in a single layer model.
sinhasam
I’m providing my code. Can you suggest my how should I go on? Although, it is in torch, I guess the idea should be similar.
Abhishek_Arya
print(lstm:getHiddenState(1)) – not sure what function will this be, but if you print out lstm and recent_hidden, it will on the last row of lstm, the values should be the same as recent_hidden. Hope it helps
chunzhu
RNNs are inherently sequential. They are auto-regressive, meaning the input for timestep t contains the output for timestep (t-1), meaning you have to first calculae the output for timestap (t-1). This is one reason why ‘attention is all you need’ is quite interesting.
hughperkins
That is assuming that the LSTM has only one layer. It isn’t the case for a stacked LSTM.
sinhasam
I experiment with using a for loop to collect every hidden state. The loss doesn’t change, but runtime is ~10x more (with a sequence length of 30). <SCODE>        embedding = self.drop(self.emb(inputs))
        
        b_len = embedding.size(0)
        outputs = Variable(torch.cuda.FloatTensor(b_len, embedding.size(1), self.hidden_size))
        for i in range(b_len):
            
            cur_emb = embedding[i:i+1, :]#.view(1, embedding.size(1), embedding.size(2))
            o, hidden = self.gru_1(cur_emb, hidden)
            outputs[i, :, :] = o
<ECODE>
Shawn-Shan
KaiyangZhou
Why not try GRU, output all the cells of LSTM will cost you huge time with step-by-step LSTMCell
gmyofustc
GRU and LSTM are essentially the same here. If you want to collect GRU hidden states, you have to loop through it as well. Only difference is GRU has 3/4 of LSTM parameters
Shawn-Shan
gmyofustc
Isn’t the last time step the last hidden state too? Sorry I’m a noob who likes to ask a lot of questions.
rchavezj
Armin_A
Salvador_Medina