Nishanth_Sasankan:

once the model reaches 99% accuracy further training for some reason changes the number of unique labels in prediction to only 1 which is label 0.


If your model reaches 99% accuracy, it starts to predict only zeros?
If so, it looks like you are dealing with an imbalanced segmentation use case, i.e. 99% of your pixel labels are 0s, thus your model just learns the majority class.
A simple function like:
def predict(image):
    return torch.zeros_like(image)

would therefore also achieve a 99% accuracy.
Could this be the case? If so, you should try e.g. class weighting in your loss function to counter the overfitting on the majority class.



 Nishanth_Sasankan:

While obtaining my label I transform it to tensor which divides each of the pixel value in it by 255. This changes my label values from [0 1] to [0 , 1/255]. I manually change this back by assigning label[label!=0]=1, followed by label=label.long().


This might work and seams reasonable, but I would recommend to remove the ToTensor() transformation for your label and just transform it manually to a tensor by e.g. using torch.from_numpy in your __getitem__.
Also be careful about resizing the segmentation target, as interpolations other than nearest neighbor will change your labels (if you are using resizing somewhere).
Your ignore_index=0 approach seems also to point to an imbalanced target distribution.
Could you count the class labels for a few target tensors using:
target.unique(return_counts=True)