The largest I could load on the GPU was 20 frames from 10 videos 128x128 each. Any sort of increase (size or number of frames) lead to CUDA out of memory error.
I tried first batch_size=1 (here 1 data sample is a vector of N frames from 1 video) with a larger number of frames and batch_size=10 with 20 frames. The former always led to CUDA out of memory error, so I decided that somehow the batch stays on GPU (I can’t find another explanation).