Thank you. The issue was indeed caused by additional RGB values in the original mask. I used the ImageDataLoader from Keras for the agmentation of the masks. It seems that for some operations it changes the RGB values in the border areas. I have now solved the problem by using torch.zeros instead of torch.empty.
Is there any advantage in using torch.empty over torch.zeros in the first place? Otherwise I would use only torch.zeros in the future to avoid these kind of problems.