I am also working on a project involving video files and I have observed the same issue as you. The way I resolved it was to disregard the idx value passed into getitem and instead do my own random indexing within the function. I randomly reload a different video 10% of the time, otherwise just keep sampling cases from it. This will greatly reduce the number of videos that need to be loaded as new video loads only happen in 10% of the sampling cases. (or less, if you like)
Pseudo code:
def __getitem__(self, index):

    if random.random() > .9:
        # sometimes reload a new video
        self.loadNewVideo()

    # select a scene, and then select a frame
    sceneIdx = random.randrange(0, len(self.videoShots))
    scene = self.videoShots[sceneIdx]
    frameIndex = random.randrange(scene[0], scene[1] - 1) 
    frames, targets = self.buildTrainingCaseFromFrameIndex(frameIndex)
    return frames, targets