I have been working on what I think should be a ridiculously simple problem.  Let’s say I have a 2D matrix 5x5.  I would like to take each 3x3 patch, flatten the patch (to 9 elements), then put the flattened patch through a network that spits out a single number.  So the 5x5 matrix should become a 3x3 matrix.  Each individual number in this resulting 3x3 matrix came from running a 3x3 patch through a network.  I’m also cool with a 5x5 matrix result if adding padding simplifies the solution.  The solutions I am coming up with create patches (using unfold) which I have to convert and then concat back together making sure the spatial relationship between the input and ouput matrix is preserved.  What am I missing here?