Facing similar issue. I compared only GPU number. Are you able to locate the source of high latency?
Another observation from my end.
If you do a dummy convolution operation with the input and carry on with the network flow. You will get correct timing for the network.
def forward(self, x):
start0 = time.time()
x1 = self.conv2d(x)
end0 = time.time()
time0 = (end0 - start0)*1000
print(‘Dummy conv2d-depth - %f’%time0)

 start0 = time.time()
 x1 = self.conv2d(x)
 end0 = time.time()
 time0 = (end0 - start0)*1000
 print('conv2d-depth - %f'%time0)

i doubt the latency spike is something related to GPU initialization in native pytorch code.

@smth  Need some pointers to locate the origin of latency increase
Thanks in advance