Thanks, we test our dataloading without training and it is very fast. But we also put some timestamp on our code and found training time is only part of total time, is that because GPU is aync and our time recoding may not be that accurate?

  
      pytorch.org
  
  
    

CUDA semantics — PyTorch 1.7.0 documentation



  
  
    
    
  
  


start_time = datetime.now()
for for loo in range(0, config['epochs']):
    for fi, batch in enumerate(my_data_loader):
        train_time = datetime.now()
        train()
        train_endtime = datetime.now()
total_endtime = datetime.now()