For anyone that may run into this in the future: I solved the “problem” by pre-splitting videos and re-saving them as pickle files of tensors that contained the relevant frames. This is completed by a custom loader that unpickles these files on loading. This adds some overhead, but its easily solved by several workers.