Could you verify that:

your model has indeed 51 output channels and returns the logits in the shape [batch_size, nb_classe=51, height, width]

the target tensors contain all class indices in [0, 50]


If so, could you try to overfit a single sample with a target containing a variety of class indices and check if the model is able to learn it?