In my experience the fallback THNN conv is much less memory-efficient than the specialized implementations of nnpack/mkldnn/cudnn etc.
As such you’re probably wasting memory if you don’t use them.
That said, you could try unfold and fold 12 with matrix multiplication. This is somewhat similar to what THNN does (at least in some cases).
Best regards
Thomas