@ptrblck: I debugged into pytorch source code to find the reason and I traced it down to the cudnn statement where the convolution descriptor is populated:
aten/src/ATen/cudnn/Descriptors.h:170
cudnnSetConvolutionNdDescriptor(mut_desc(), dim, pad, stride, upscale, CUDNN_CROSS_CORRELATION, mathType);
(https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/cudnn/Descriptors.h#L170 5)
When changing the mode from CUDNN_CROSS_CORRELATION to CUDNN_CONVOLUTION, it will work and it will use tensor cores. I assume this is a bug inside cudnn, since both modes should work equally well, just like they do in 2D.
I verified this effect using my completely separate C++ code which uses cudnn directly. Just by changing the convolution mode, I can see tensor cores turning off or on, keeping all other settings.
EDIT: So when I used my custom pytorch version with CUDNN_CONVOLUTION, I still needed to modify the above example code slightly, because the convolution workspace was to large. But if you change the batch size from 8 to 1 it should work.