Suppose I have a video dataset which is containing human’s facial expression videos. Each video is corresponding to one expert-labeled-emotion-score (which is a number between 0-10 indicating the intensity of the emotion for the whole video). So, I’d like to train a deep learning model on this dataset using the CNN-LSTM architecture. However, I’m worrying about which time-step in the LSTM output that I need to decode for computing the loss in order to optimise the model’s parameters. Normally, we decode the last output of LSTM, since the network takes the information in the past to predict the current input. However, in my case, the label score is for all frames of the video and therefore, I think decoding the last LSTM output wouldn’t solve the problem. Could you give me some suggestion to solve this problem ?
The number of frames in each video ranging from 150-700 frames.