do you have cudnn benchmarking mode turned on? (torch.backends.cudnn.benchmark=True in python, not sure about C++ apis). If so, pytorch would search for the fastest convolution algorithm for each input size and that can slow you down.