Hi,
I made the following observation which I can not follow. I am using the deeplabv3_resnet50 model from torchvision and run it in eval mode with different batch sizes. The runtime including a torch.synchronize() are almost linear with the batch size. That means that the rate of images/second is almost constant. I also tried out different settings of cudnn.benchnmark = True/False. I was hoping to see a constant inference time per batch over all batch sizes because they are running in parallel on the gpu. Am I getting anything wrong here?
2 - 0.02s
4 - 0.031s
8 - 0.05s
16 - 0.094s
32 - 0.178s
Thanks for your help.
Markus
torch=1.4.0
CUDA Version: 10.0
Tesla V100