This is specific to CPU – the CPU convolution code is not very optimized (for memory usage) and uses a lot of memory. The CUDA convolution implementation (via cuDNN) is much faster and uses less memory.