Instead of to_tensor you could create the mask tensor via torch.from_numpy().
For 3 classes and a background class, I would assume your model should use nb_class = 4, which would most likely correspond to the number of output units.