I’m trying to optimize the inference latency, but found that there’s a strange overhead happens in one single conv layer. Even though it still fits y=kx+b relationship, but the b is unacceptable large. How should I reduce such b? Thanks in advance!
The profiling result goes as follows.




[filter, filter, input_channels, output_channels]
Latency




[3, 3, 16, 8]
0.024658


[3, 3, 16, 16]
0.032011


[3, 3, 16, 32]
0.031948


[3, 3, 16, 64]
0.037025


[3, 3, 16, 128]
0.049538


[3, 3, 16, 256]
0.062251


[3, 3, 16, 512]
0.105888



The code for profiling goes as follows:
for i in range(7):
  shape = [3,3,16,2**(i+3)]
  kernel_value = np.random.rand(shape[0],shape[1],shape[2],shape[3]).astype(np.float32)
  kernel = torch.as_tensor(np.transpose(kernel_value, (3,2,0,1)))
  input_value = np.random.rand(1,16,32,32).astype(np.float32)
  x = torch.as_tensor(input_value)

  before = datetime.datetime.now()
  for j in range(100):
    if j==50:
      before = datetime.datetime.now()
    tmp = F.conv2d(x, weight=kernel,bias=None,stride=1,padding=(3-1)//2)
  after = datetime.datetime.now()
  interval = after-before
  print(str(shape)+"\t"+str(get_seconds(interval)))