This was very helpful @ptrblck & @vdw ! 
So the output is basically the collection of the hidden states’ step-wise. I get it now.
So when papers say that they use the bidirectional embedding of a sentence in PyTorch, they mean they use the final "hn"i.e “forward_final_hidden_layer, backward_final_hidden_layer” right?
because using the output[-1] is “forward_final_hidden_layer, backward_initial_hidden_layer”, which is incorrect to use as a sentence representation, correct?