Thank you for the input!
I tried swapping out the Conv2dTranspose with an Upsample2D/Linear Interpolation and I’m seeing the same artifact.  :’(
The network is pretty much all Conv2ds, so I don’t think I can really ‘remove them’?  And WRT overlap,  I thought that kernels have to be able to overlap for convolution to be useful?  There are some articles on it that seems to imply it’s okay: Deconvolution and Checkerboard Artifacts 1
Other things I thought might have been problems: BatchNorm with bias=False.  Normally for overfitting the bias channel is the thing that takes up the brunt of the effort.  By setting bias to false it forces all of the image data to get encoded in the weights, which for a zero input means that there’s no chance of a decent fit.  I turned off batch norm and it didn’t seem to help.
My next step to debug is going to be replacing the all-zero input with all-one input.  I’m not going to put back the batch norm yet.
If that fails, I’ll try pulling everything but the U part of the network.  I think the depth-convolution might be working on pixel channels instead of depth.  :c  Doesn’t seem like it, but…