Right - slicing the image up into “center” and then a set of rows and columns is one option. The other is to change the kernel for those “pixels” and keep the image in a single buffer. Specifically, we need to adjust the kernel by renormalizing it after excluding values “outside” the source buffer, rather than by assuming those values are zero and using them with a constant kernel as in tf.conv2d. Wikipedia calls this approach “kernel crop”. Note that “extending” the edges is also acceptable in most situations with image processing, and is often easier to implement on the CPU. The “wrap” and “mirror” options are more specific to texture rendering in 3D.
Assume a 5x5 kernel for simplicity (it should always be an odd width). Ignoring the corners for now, the first row would want a 3x5 kernel, renormalized after removing the two rows that are outside the source buffer. The second row would require a renormalized 4x5 kernel.
Note that the GPU hardware supports the “extend”, “mirror” and “wrap” modes in the core texture sampling hardware. I don’t know how this maps to the various TF backends, but I’m curious to learn. The hardware also supports non-integer bilerp sampling, which would also help, but that’s another question. So, in addition to the slicing you propose, we can also pad the input by extending the final row and column out to half the kernel width.
I’m happy to dive in and learn these items and was mostly looking for tips as to where to start as this is my first spelunking into the building of models from scratch in TF.