The typical solution is to group the videos by length to get buy with as little padding as possible. There are various methods of implementing this “bucketized” sampling, TorchText someting like it, I sometimes implement something like it by grouping them in the dataset and then returning batches of data from the dataset (and using batch size 1 in the dataloader and squeezing the extra dimension).
Best regards
Thomas