Solved by ptrblck in post #4 
              
              
                cudnn might use different kernels for different setups (input shape, conv setup). In your example, a matrix multiplication kernel should be used and on my machine these kernels are used: 
CUDA Kernel Statistics:

 Time(%)  Total Time (ns)  Instances   Average   Minimum  Maximum                     …