Thanks for your help, now everything is clear for me!
Also I run your script at Tesla V100 at AWS, you could add this values to your page:
cuda version= 9.0.176
cudnn version= 7003
pytorch's vgg16 eval at fp32: 26.2ms avg
pytorch's vgg16 train at fp32: 83.5ms avg
pytorch's vgg16 eval at fp16: 12.6ms avg
pytorch's vgg16 train at fp16: 58.8ms avg
pytorch's resnet152 eval at fp32: 38.7ms avg
pytorch's resnet152 train at fp32: 136.5ms avg
pytorch's resnet152 eval at fp16: 21.7ms avg
pytorch's resnet152 train at fp16: 92.9ms avg
pytorch's densenet161 eval at fp32: 48.3ms avg
pytorch's densenet161 train at fp32: 142.5ms avg
pytorch's densenet161 eval at fp16: 35.7ms avg
pytorch's densenet161 train at fp16: 102.3ms avg