I highly recommend to you to work with the video files itself.
First of all, you do not need a custom DataLoader you only need a custom Dataset. Check out this tutorial 92. I like to use cv2 module, just do conda install opencv or pip install cv2.
in the Dataset's __getitem__() method you can just read a frame like following
MyDataset(Dataset):
    '''Custom dataset-class for loading video-frames'''
    def __init__(self, num_videos, video_directory, videos):
        self.num_videos = num_videos
        self.video_directory = video_directory
        self.videos = videos

    def __getitem__(self, i):
        cap = cv2.VideoCapture(os.path.join(self.video_directory, self.videos[i]))
        
        # reading all frames into a big ndarray
        width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)
        height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
        num_frames = cap.get(cv2.CAP_PROP_FRAME_COUNT)

        frames = np.zeros((num_frames, h, w, 3))

        count = 0
        while count < num_frames:
            succesful, frame = cap.read()
            if not succesful:
                break
            else:
                frames[count] = frame

        labels = # get the labels for this video from somewhere
        return {'frames': frames, 'labels': labels}

This is how you can get all frames from video files into a numpy array, to convert each frame to a torch.image you will have to transpose its axis: frames.transpose(3, 0, 1, 2) where the new shape is (channels, num_frames, h, w) because torch.image needs channel before d, h, w, where d is the depth of your video aka the sequence_length if you will be using RNNs.
With the returned dictionary of MyDataset you can, while training, iterate the dataloader and call the outputs by their keys.