RAM and GPU memory are pretty much half free. They don’t seem to increase overtime. Also the only thing I do in the dataloader is load a lz4 archive and modify numpy arrays.
The freezing happened to me before, just because the ram was using swap memory, this doesn’t seem to be the case though.
Going from 3 workers and batch 128 to 4 workers and batch 32 doesn’t seem to help with workers being timed-out.