Yeah, you are right. Forget my gist. It’s way to complicated. 
Somehow I got stuck thinking about how the “window” moves along the input volume and made the code too complicated.
What times do you get? Is the conv still faster on your machine? I’ve updated my code with your approach and while the linear approach for the 1D case is faster than conv, it seems to be the contrary for the 2D case.