@gphilip
Thanks for your remind.
To verify your guess, I modify MyConv. (See the previous post.)
After modification, the memory usage is proportional to the number of layers and it is larger than the default Conv.
when set use_defualt_conv=True, the output is
memory for in-place mode: 819.6
memory for out-place mode: 1619.6

when set use_defualt_conv=False, the output is
memory for in-place mode: 2401.6
memory for out-place mode: 2417.6

Those results suggest that in-place mode of Relu doesn’t work for custom autograd functions.
In fact, my real custom function is more complex than the default Conv. I just provide an easy example to explain the problem.