Thank you very much Chris for your reply.
I think I have got the solution.  But to be clear, please check whether I am right.
The problem was, I was expecting 2 final hidden state as the batch size is 2 and I assumed that lstm taking the input as (batch, sequence, embedding) but I noticed that that is not the default case. Lstm takes input as (sequence, batch, embedding). So I have to make the batch_first=true
The output I got,
hidden[0]:
tensor([[[-0.0538,  0.1163,  0.0256],
         [ 0.0165,  0.1825, -0.0107]]], grad_fn=<StackBackward>)

Which I was expecting.