Now i change to the same cuda and cudnn version. the results are similar as before. The network is standard torchvision resnet50 model. i also incoude the profiler chrome tracking
here are the results:
using pytorch1.6
1.6_tracing1647×308 61.7 KB
output_nr                    0.00%            1.705us          0.00%            1.705us          1.705us          0.00%            2.048us          2.048us          1                []                                   
is_leaf                      0.00%            1.823us          0.00%            1.823us          1.823us          0.00%            1.023us          1.023us          1                []                                   
is_leaf                      0.00%            1.717us          0.00%            1.717us          1.717us          0.00%            1.024us          1.024us          1                []                                   
max_pool2d_with_indices      0.05%            486.468us        0.05%            486.468us        486.468us        0.17%            1.756ms          1.756ms          1                []                                   
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 987.386ms
CUDA time total: 1.046s

pytorch
	torch.__version__              = 1.6.0.dev20200516+cu101
	torch.version.cuda             = 10.1
	torch.backends.cudnn.version() = 7603
		torch.backends.cudnn.benchmark      = False
		torch.backends.cudnn.enabled        = True
		torch.backends.cudnn.deterministic  = False
	torch.cuda.device_count()      = 1
		torch.cuda.get_device_properties()  = _CudaDeviceProperties(name='TITAN X (Pascal)', major=6, minor=1, total_memory=12192MB, multi_processor_count=28)
		torch.cuda.memory_allocated()       = 0 GB
		torch.cuda.memory_reserved()        = 8 GB

using pytorch1.4
1.4_tracing1619×381 63.4 KB
cudnn_convolution            0.16%            842.643us        0.16%            842.643us        842.643us        0.98%            8.491ms          8.491ms          1                []                                   
add                          0.00%            24.037us         0.00%            24.037us         24.037us         0.00%            9.215us          9.215us          1                []                                   
batch_norm                   0.06%            335.818us        0.06%            335.818us        335.818us        0.15%            1.270ms          1.270ms          1                []                                   
_batch_norm_impl_index       0.06%            329.220us        0.06%            329.220us        329.220us        0.15%            1.268ms          1.268ms          1                []                                   
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 534.585ms
CUDA time total: 863.694ms

accuracy 0.687500

pytorch
	torch.__version__              = 1.4.0
	torch.version.cuda             = 10.1
	torch.backends.cudnn.version() = 7603
		torch.backends.cudnn.benchmark      = False
		torch.backends.cudnn.enabled        = True
		torch.backends.cudnn.deterministic  = False
	torch.cuda.device_count()      = 1
		torch.cuda.get_device_properties()  = _CudaDeviceProperties(name='TITAN X (Pascal)', major=6, minor=1, total_memory=12192MB, multi_processor_count=28)
		torch.cuda.memory_allocated()       = 0 GB
		torch.cuda.memory_reserved()        = 8 GB


again, all code, results can be find at : https://drive.google.com/drive/u/0/folders/1vyTkqBpQwUvCSnGjjJPdzIzLTEfJF8Sr 2