your speed is 129 images/sec which is lower than I expect see https://developer.nvidia.com/deep-learning-performance-training-inference 5 the pytorch resnet50 model does inference at 905 images/sec on a single V100. I will try increasing batch size as resnet18 is fairly small mode for a V100 and number of workers for dataloader.