Thanks for the update. If you think that the sides might be “different” in any sense, could you add a random flip transformation to the training, which should hopefully get rid of these artifacts.
Let me know, if that helps.