Can you provide any more info on the dataset/labels, but it seems that with a sequence of videos and only 1 datapoint of the score for the entire scene, I would guess your best bet is to return all the hidden states (and get a estimated score for each one) and take the average over that to get a result.