(I think you meant to alert @ptrblck ; he is the one who asked for the code.)
Thank you for the code. After playing with it for a bit I discovered something interesting: If I set use_default_conv=False and change
for i in range(50):

to
for i in range(50000):

then the run takes much more time to complete (as expected), but the memory use reported still remains at 48.0.
This suggests to me that the number of floats in MyConv is too small to cause any real gains when using ReLU in place, even when there are a 50000 copies of MyConv in the net. This may be the explanation.