Hi, I am using a batch size of 640 on the following model:

  
      
      GitHub
  
  
    

rosinality/relation-networks-pytorch

Relation Networks for CLEVR implemented in PyTorch - rosinality/relation-networks-pytorch


  
  
    
    
  
  


It’s a simple CNN with a relation network. Most of the computation time is spent merely on loading the data. In fact, I checked the GPU usage and most of the time they’re not being used. I have 4 GPUs and they seem to process the data quickly, once it’s loaded. I also have 40 CPUs serving as workers.
However, I am in a hurry and the training time is too long with my current setup (3 weeks!). The bottleneck is the data loading I believe. See the __getitem__ and collate functions in the dataset.py of the link above.
Is there any way to speed up data loading with such large image batches? Perhaps a way to leverage my 40 CPUs? Any help would be greatly appreciated; this is particularly urgent.