The original segmented image returned in dataset(w/o ToTensor Transform) is a segmentation mask having a value of 0-20(category) or 255(background) for each pixel . But with ToTensor() Transform It will normalize the mask (dividing by 255) and But to use it for cross entropy loss i need to denormalize and in addition i have replaced the values of background (255) with extra category (21). Segments seems to contain right value , I have tried Overfitting a small dataset ,Tuning learning rate etc also , But It seems there is some basic flow in set up that loss is not decaying .