torch.utils.bottleneck might be rerunning to profile the CUDA kernels. How many times is the script being rerun?
To narrow down a potential data loading bottleneck, you could also use this AverageMeter object 14 as used in the ImageNet example.