So as I understand your use case you would like to classify each frame separately.
The video sequence is therefore not important, i.e. each frame has the target of the video target?
If so, you could just load the video, e.g. with OpenCV, and save each frame of the video into separate folders.
Using these images, you could then train a supervised model on the data.
I would split the videos in a stratified fashion, i.e. approx. using the same class distribution for training, validation and test data.