I get that the emotion might change per frame, but if the only label you have is for the entire video these are the only two option I see, since the frames themselves are unlabeled.
The goal of your lstm should be to determine the emotion-score given the sequence/video, so if you provide the entire video (given you can fit it all within your model) the lstm should provide the estimated emotion score as it’s output.
I might be incorrect, but you aren’t using the ‘last frame’ for the output per-say, instead you are getting the last hidden state (over the entire sequence) and using that to determine a final estimate, so if the sequence is long enough (i.e. has some of the ‘highest score’ frames’) it should generalize to the expected score.