Since mask_out is created as an empty tensor here: torch.empty(h, w, dtype=torch.long), it will contain uninitialized values at the beginning.
The loop over self.mapping should assign valid class indices to the mask, which doesn’t seem to be the case and some indices are still uninitialized.
I would recommend to recheck the mapping and make sure that only the RGB values in self.mapping are really used in the mask.
You should be able to get all unique color values via:
mask = torch.zeros(3, 24, 24)
mask[:, 0, 0] = torch.tensor([1., 2., 3.])
mask[:, 1, 1] = torch.tensor([4., 5., 6.])
mask[:, 2, 2] = torch.tensor([7., 8., 9.])

print(mask.view(3, -1).unique(dim=1))
> tensor([[0., 1., 4., 7.],
          [0., 2., 5., 8.],
          [0., 3., 6., 9.]])