I have not used transforms for my  image , as I have a very large dataset. But if I am using transforms, the same transforms have to be applied to image and mask. Since you are using random rotations and flip, make sure your image mask pair g through the same random transforms.
i:e if ImageA is randomly rotated by 10 degrees, make sure MaskA is also rotated by 10 degrees.