Depending on the algorithm you use to perform the convolution, the intermediary results can be very (very) big. In particular, if you do it as a matrix multiplication, the unfolded Tensors will be huge here.
The reason I recommend benchmark is that it will adapt the algorithm to the inputs instead of simply using the default one. This should help you get lower memory (and better speed after the first call ran the benchmark).