Since you are using two output channels in your model and nn.CrossEntropyLoss, you would have to apply a mapping between the colors of your target maps and the class indices.
If your current targets contain (normalized) RGB values, label would be a tensor full of ones.