You could try a UNet-like architecture without the downscaling, but with a similar arrangement of skip connections (i.e. have one from the first to the last, one from second to one-but last). You probably don’t need very many layers…
Best regards
Thomas