Soo I’m gonna give you some hints. I’m gonna assume your coding skills are more or less high.
The first one is not to use opencv. It doesn’t really work well.
Suggestions from my side from better to worse:

Convert the videos to numpy arrays if you have enough hard disk. Use memory map.
If not enough hard disk try using Nvidia Dali. It loads video by using GPU. It’s the the fastest dataloader you can use but it’s not super flexible.
If you want to have more flexibility but still quick loading try GitHub - JuanFMontesinos/PyNVIdeoReader: GPU Accelerated video decoder 13, it’s a video reader I wrapping ffmpeg and with gpu capabilities.
If none of the previous fits your expectations, use imageio or a ffmpeg wrapper.

The bottleneck is reading the videos, that’s what you have to optimise.