I’ve seen the strange padding and tried to reproduce the error with a “standalone” CNN using your conv layers, but failed to do so.
Good it’s working now, but something is still going on in your original architecture.