I try to train 3d conv net for video recognition. My expected data volume per batch is [32, 3, 32, 224, 224]. When I use DataLoader, it seems so slow for such a huge volume. I wait ~300 secs for starting to serve batches with 8 workers. And general scheme of timing is ~300 sec wait for one iteration and then no wait for next 4 learning iterations and then wait 300 secs and so on. I think it requires 300 secs to ready all batches in all workers but clearly it is away from practical. I try to increase workers then wait time increased for jammed iterations.
Note that, I’ve enough memory and It does not overload the memory. One concern is, I use HDD and lookup time is large undoubtedly. Do you have any suggestion to improve data loading scheme ?