Thanks. I tried increasing the num_workers and it seems my bottleneck isn’t the data loading.
As far as I understand this, I can’t infer without DataParallel if I have trained the model on multiple GPU’s? It was throwing errors in loading the model state_dictionary.
I will check out the torch.cuda.synchronize.
Thanks. Also, The model I am using is a modified ResNet50. Not the shallowest of networks.