Yea this was my first step. I have made my custom data loader and now I think I need to make the CNN Encoder and LSTM Decoder for the pipeline.  Here is the code so far:
import datasets
import transforms
import video_csv_creation
import torch
import torchvision

video_folder_path = ''
csv_save_path = ''
data_loader_save = ''
video_csv_creation.generate_video_csv(video_folder_path,csv_save_path)

# CUDA for PyTorch
use_cuda = torch.cuda.is_available()
device = torch.device("cuda:0" if use_cuda else "cpu")
torch.backends.cudnn.benchmark = True

# Parameters
params = {'batch_size': 2,
          'shuffle': True,
          'num_workers': 4}

dataset = datasets.VideoLabelDataset(csv_save_path,     
        transform=torchvision.transforms.Compose([
            transforms.VideoFilePathToTensor(100, fps=15, padding_mode='last'),transforms.VideoResize([256, 256]),
        ]))
        
data_loader = torch.utils.data.DataLoader(dataset, **params)
torch.save(data_loader,data_loader_save)
print("Dalatoader Saved")
for videos, labels in data_loader:
    print(videos.size(), labels)


Now the data loader gives a tensor of - size batch_size x channels x frames x height x width  and then the label something like-
torch.Size([2, 3, 50, 256, 256]) tensor([1, 1])
torch.Size([2, 3, 50, 256, 256]) tensor([1, 0])

Now how can I make a CNN which takes in a batch of 2 consisting of 50 frames of 3 channels each and then labels for both of the frame groups? So far I have only worked with CNN’s which takes batches consisting of frames and respective labels. Thanks.