Are there different emotions in the videos or just videos of one emotion with a single score?
If its the former you can use one-hot encoding for the emotion and the magnitude of the encoding for the emotion-score.
But I agree, talking the average isn’t an optimal solution but given the lack of labels I’m not sure how else you could approach this.