Hmm yeah I guess that would work! unfold seems to indeed return a new tensor instead of a view, so I guess it can use quite a bit of memory (esp. for 2D and higher convolutions).
I’ll do some benchmarks vs. regular convolutions. Thanks @ptrblck!