Solved by Naman-ntc in post #2 
              
              
                You should look at torchvision functional transforms 
You can randomly choose augmentation stratergy and apply on both input and ground truth