Abhishek_K:

As last few time steps will have more of padding vectors. Hence you have suggested to take the final hidden state rather than taking hidden state for each time step. Hope my understanding is correct?


Yes, hidden contains the last hidden state w.r.t. to the last time step, even if you use packing where you have sequences of different lengths. So if one sequence is of length 20 and the other of length 15, hidden will contain the 20th and 15th hidden states since both are the last hidden states for these to sequences.



 Abhishek_K:

Yes, I have taken care of the view part. I am using view only in the final step , to reduce the tensor size from ( batch_size x 1(no.of layers) x hidden size) → (batchsize, no.of layers * hidden size). Could I have used squeeze instead of view to reduce the dimension of the tensor?


The challenges is not to get the shapes right, but to also ensure that the data does not get scrambled up in the process :). I always consult the documentation to be sure:


h_n of shape (num_layers * num_directions, batch, hidden_size) […], the layers can be separated using h_n.view(num_layers, num_directions, batch, hidden_size)





 Abhishek_K:

I tried applying a non linear activation ( ReLU) after computing 1 Liear Layer or tried applying it on the hidden layer of LSTM. But doesnt seem to have any effect.


The heavy lifting in terms of learning is probably done in the LSTM layer, so having 1, 2 or 3 additional linear layers might not be that important. This also means that you would see a big effect be removing/including a non-linearity. However, I would generally always include one between 2 linear layers.



 Abhishek_K:

I am intializing the hidden layers after running every batch. […] Could you kindly give your inputs on this regard too?


Re-initializing the hidden state after/before each batch is the standard approach. It also in a sense meaningful if your batches are independent. Say you have independent short texts such as tweets, why should one tweet start with a different initial hidden state then another tweet. Of course, if you build your batches so that the sequences are dependent – e.g., for a language model where use a sliding window over a long text to create your (sequence, target) pairs – then yes preserving the hidden state across batches makes sense.
Colab throws a memory error because the computational graph for hidden will only get larger, so backpropagation will take longer and longer. If you want to preserve the hidden state between batches, you can use hidden.detach() (I’m not sure about the exact syntax). It detaches the hidden state from the computational graph. So it’s kind of a new hidden state but with the same values as the last one.