PyTorch 1.2 CUDNN 7602, cudnn.enabled=True ==> OK
PyTorch 1.3 CUDNN 7603, cudnn.enabled=True ==> OK
PyTorch 1.5 CUDNN 7603, cudnn.enabled=True ==> NOT OK (slow forward)
I’ve tried those configurations… seems that the problem is not in CUDNN but somewhere in torch 1.5. Actually I perform BatchNormalization after each layer so there are more than one BN layer, here is the model:
DGCNN(
(conv1): Sequential(
(0): Conv2d(6, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): LeakyReLU(negative_slope=0.2)
)
(conv2): Sequential(
(0): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): LeakyReLU(negative_slope=0.2)
)
(conv3): Sequential(
(0): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): LeakyReLU(negative_slope=0.2)
)
(conv4): Sequential(
(0): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): LeakyReLU(negative_slope=0.2)
)
(conv5): Sequential(
(0): Conv1d(512, 1024, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): LeakyReLU(negative_slope=0.2)
)
(cls): Sequential(
(0): Linear(in_features=2048, out_features=512, bias=False)
(1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): LeakyReLU(negative_slope=0.2)
(3): Dropout(p=0.5, inplace=False)
(4): Linear(in_features=512, out_features=256, bias=True)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): LeakyReLU(negative_slope=0.2)
(7): Dropout(p=0.5, inplace=False)
(8): Linear(in_features=256, out_features=40, bias=True)
)
)
I’ll keep investigating in profiling the batchnorm layers forward. The problem is not in cudnn benchmark, this is not the first forward and I also tried to disable it.
Actually I’m stuck with PyTorch 1.3 and some libraries working only with torch > 1.4…