I ran the resnet18 imagenet example on an AWS p3.2xlarge on the official amazon AMI for deep learning, and I’m finding that the data-loader is a bottleneck, evidenced both by a lack of constant GPU utilization and also the fact that when allowing the training loop to run with a constant data (no iteration through the dataloader), the time to train an epoch is cut by nearly 50% (and full GPU utilization). Has anyone else experienced this? If you want to replicate the issue exactly, replace data loader loop with a “for i in range(len(dataloader))” and set data to be some constant batch of data so that no data is actually required to be read from disk.