Suppose I have converted videos to frames and the folder structure is such

So now I can use the ImageFolder to load the frames per the classes. But training models frame-wise seem to perform very poor for me. So I want to stack all the frame per video and assign a single label against it. What would be an easy way to achieve this?