If I perhaps pre-process the data into the following structure, it would make life a bit easier?
Downside of this approach is limited flexibility to shuffle and do things like cross validation.
- train_data
    - class_0
         - vid1_0
              - frame0
              - frame5
              - frame10
              ...
         - vid1_1
              - frame1
              - frame6
              - frame11
              ...
         - vid1_ ...
              ...
         - vid2_0
             - same structure as vid1
         ...
     - class_1
         - same structure as class_0

- val_data
    - same structure as train_data, in this instance