What sort of network are you applying to the 3x3 patches?  What you’re describing is effectively 2D convolution, but instead of a linear/affine operator applied to the patch you’re applying a network operator.