This might be helpful to you Video Dataset Loading in PyTorch 72.
Its a custom dataset implementation that goes very well with any video dataset and is very fast. It also allows you to specify the amount of frames to load from each video, taking the frames evenly from start to finish of a video. This way you might be able to make your loaded frame tensor a bit sparser in terms of the amount of frames it contains from the video, and allow yourself to not overrun memory