Your concern is correct assuming the mask contain class indices. In these cases transformations such as resizing can still be applied to a mask tensor but the nearest mode should be selected so that class indices are kept and are not interpolated. Often if a wrong transformation is used float values will be created in the mask, which would then fail during the loss calculation as e.g. nn.CrossEntropyLoss expects LongTensors for the target. In any case, I would expect official examples as well as popular repositories to take care of these mistakes.