You might want to checkout SequentialSampler, which is related to data loader: https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html 70
My experience with SequentialSampler is that it will keep your training data sequence, so that the video frame sequence can be reserved during training.