Hi,
TLDR; “Conv2d uses tensor cores, Conv3D doesn’t when using apex AMP or FP16”
According to NVIDIA, cudnn 8 does support tensor core operations for 3D convolutions.
To be sure if tensor cores are really being used (HMMA instructions) I am checking this with the nvidia profiler with the sm__inst_executed_pipe_tensor_op_hmma.sum metrics activated (See https://developer.nvidia.com/blog/using-nsight-compute-nvprof-mixed-precision-deep-learning-models/ 13)
I am really up to date by building pytorch from source:
My setup:
PyTorch: 1.7.0a
cudnn: 8.0.1
CUDA: 10.2
GPU: RTX 2080 Ti
Simple test script:
import torch
from torch import nn
import torch.nn.functional as F
from time import time
import logging
from apex import amp

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Conv3d(64, 64, 5, padding=2)
        self.pool = nn.MaxPool3d(2, 2)
        
    def forward(self, x):
        x = self.pool(F.relu(self.conv(x)))
        return x


if __name__ == '__main__':
    net = Net().to("cuda")
    amp_net = amp.initialize(net, opt_level="O1")

    input_image = torch.rand((8, 64, 64, 64, 64), device="cuda")
    gpu_net = amp_net

    gpu_net(input_image)

Behavior:
If I change the network to 2D input and 2D convolutoins, I can see that tensorcores are used in the metrics and that there is a speed-up.
For 3D they are not being used.
EDIT: Instead of using AMP, I also tried converting the net and inputs to FP16, which also worked in the 2D case but not in 3D
Maybe @ptrblck can help?