No, Each mask has at max 4 classes(Red,blue,green and black), I did what you told apart from the last statement loss = torch.mean(loss) because I am calculating the average across batch using train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss)). I was encountering a error because my make_weight_map(masks) was expecting a numpy array instead of tensor and also I forgot to move tensor to cpu ,which I did now. but now, I guess my system has kind of stalled I guess.
Edit:I got the memory error now though my ram size is 32 gb.