Thank you for your reply
But the conv op written by my hand does not blow up the memory,
x = w.mul(x).sum(-2)
x = x.view(N,F,H_out, W_out)

it works just fine. Larger than the nn.Conv2d, yes but not larger so much.
test again it’s 940M v.s 1500M in nn.Conv2d and my conv.
And the result has x1e-4 difference. :  (sim(x)-conv(x)).sum() < 1e-4 * decay_rate if x = torch.rand(2,64,300,300) * decay_rate
Yes, my conv has some difference with the nn.Conv2d(I don’t know why), but the result is similar.
And I just added two operators in the code above QAQ!!
x = x.sub(t).abs().mul(w).sum(-2))
x = x.view(N,F,H_out, W_out)


Aha, I just changed the sub(t) to sub(0) in there
x = x.sub(0).abs().mul(w).sum(-2))
x = x.view(N,F,H_out, W_out)

it works just fine, similar memory with my conv
So, it must be the sub(t) caused this problem
BTW, t is a parameter with the same size with w