Solved by ptrblck in post #6 
              
              
                If you would like to classify each video sequence (64 frames) to a single class (binary classification), your output and target should both have the shape [batch_size, 1]. 
To achieve this you would need to reduce the model’s output, e.g. using an nn.Linear layer as the final classifier.