@albanD Btw when I set torch.backends.cudnn.benchmark = False
I still get 3 calls to FFT algorithm (1 of the 4 calls) as shown in the following snippet from CUDNN logs which are a lot fewer (vs. 39) than using torch.backends.cudnn.benchmark = True. What could be causing it?
I! CuDNN (v7601) function cudnnConvolutionForward() called:
i!     handle: type=cudnnHandle_t; streamId=(nil) (defaultStream);
i!     alpha: type=CUDNN_DATA_FLOAT; val=1.000000;
i!     xDesc: type=cudnnTensorDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[18432,6,14,14];
i!         strideA: type=int; val=[1176,196,14,1];
i!     xData: location=dev; addr=0x200154c00000;
i!     wDesc: type=cudnnFilterDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         vect: type=int; val=0;
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[16,6,5,5];
i!         format: type=cudnnTensorFormat_t; val=CUDNN_TENSOR_NCHW (0);
i!     wData: location=dev; addr=0x200118400a00;
i!     convDesc: type=cudnnConvolutionDescriptor_t:
i!         mode: type=cudnnConvolutionMode_t; val=CUDNN_CROSS_CORRELATION (1);
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         mathType: type=cudnnMathType_t; val=CUDNN_DEFAULT_MATH (0);
i!         reorderType: type=int; val=0;
i!         arrayLength: type=int; val=2;
i!         padA: type=int; val=[0,0];
i!         strideA: type=int; val=[1,1];
i!         dilationA: type=int; val=[1,1];
i!         groupCount: type=int; val=1;
i!     algo: type=cudnnConvolutionFwdAlgo_t; val=CUDNN_CONVOLUTION_FWD_ALGO_FFT (4);
i!     workSpace: location=dev; addr=0x2001c0000000;
i!     workSpaceSizeInBytes: type=size_t; val=679486848;
i!     beta: type=CUDNN_DATA_FLOAT; val=0.000000;
i!     yDesc: type=cudnnTensorDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=**CUDNN_DATA_FLOAT (0)**;
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[18432,16,10,10];
i!         strideA: type=int; val=[1600,100,10,1];
i!     yData: location=dev; addr=0x2000a3800000;
i! Time: 2020-01-16T12:23:07.839722 (0d+0h+0m+25s since start)
i! Process=86375; Thread=86375; GPU=0; Handle=0x13e4302b0; StreamId=(nil) (defaultStream).