I was wondering the same as I have seen examples (for text classification) that use the last output (output[-1]) as well as the hidden state. See https://github.com/jiangqy/LSTM-Classification-Pytorch/blob/master/utils/LSTMClassifier.py and https://github.com/prakashpandey9/Text-Classification-Pytorch/blob/master/models/LSTM.py. I have also tried a siamese network with contrastive loss as well as MSE on both the output and the hidden state.
Lessons learned: when I built my lstm classifier I used the output and got accuracy around 98. With the hidden - not that good (don’t remember the scores). But interesting was the Siamese network - with MSE and output - great results. With contrastive and output -very bad results. MSE and hidden - bad results; contrastive and hidden - bad results. So basically using the hidden to compute a similarity score or the output has a huge difference in this scenario. However, it also depends (for my case) on the loss function.
Hope this gives some more clarity.
Cheers,
Dimitar