What format is the video to be fed in?  frame by frame so images?  You can write your Dataset class to handle annotated and unannotated images and “ignore” those frames… I would keep annotated and unannotated images in totally separate Dataset instances myself… you can see in the example I pasted they seem to just create two different loaders and datasets from the same core data (plus some transforms on the training set)… you could easily just slice off all non-annotated frames using a collate, a transform… or just use some intelligence in the Dataset class itself to “drop” data which you cannot use for training