Ok, so as far as I understand, the operation is equal to a convolution regarding the size, stride etc., but instead of the dot product, you would like to perform some other operation.
Tensor comprehensions might work, but currently doesn’t support padding 3.
What do you think of using im2col as a potentially ugly hack?
Would that work for your function?
Have a look at Pete Warden’s blog post 24.