Hey seems I misunderstood your question in my last reply, you have only 1 label (B,1) for each video, but you can extend the label for the entire video (since the score if for the entire video it should be okay to assume that the score is valid for a single frame in that video).
Similarly it should also be okay to only take the last output of the lstm since it should give you the score for the entire sequence (which is the label for the video you inputted into the lstm)