Thank you for your reply. So I applied interpolation NEAREST on both the images and masks in the dataset and I added a padding of 1 in the double convolutions, so that now I can have same input and output size ( I changed the images to 256*256). Also, I have changed the loss function to BCEWithLogitsLoss() as you suggested. Now the training process seemed fine in terms of numbers, but the results are not correct.
res508×701 123 KB  For example, these are the results when loss is 0.015270. Any more ideas what is wrong?
Thanks again!