There are a variety of convolution algorithms.
Some are calling an (implicit) im2col and use a matrix multiplication afterwards. Other might use an FFT  or Winograd approach etc.
Also, if you are using torch.backends.cudnn.benchmark = True, the first iteration for a new input shape will run some benchmarking and select the fastest kernel.