For the task of inpainting, various loss functions have been proposed till now. In the images shown, the black box artifact is present in the area where image data is already available. So ideally you should not get that kind of artifact. I would be interested to know the loss function used. If it was simple L1 between the whole predicted image and gt image, I believe this can happen. One way to handle it using loss function is you calculate loss only for regions which are being filled. So that data which is already available can be used. Something like (mask*predicted) - (mask * GT).