Refer the docs of torch LSTM how the input data should be arranged. It is an excellent piece of documentation. Kind of felt like the input data is not arranged properly in the code. Definitely use the batch_first = True option in the lstm. Makes life a lot easier.
Docs : LSTM_Pytorch 1