Solved by ptrblck in post #7 
              
              
                Spatial transformations applied on the input image should also be applied on the mask tensor to make sure that the input pixel location still correspond to the mask (e.g. rotaions, resizing etc.). 
Besides that you should treat the mask as a classification target and should not change its values.