You can use the print(mask..view(3, -1).unique(dim=1)) operation to print all unique RGB values in your masks and make sure that your dict used to map RGB values to class indices indeed contains all unique colors.
Currently I guess your dict is missing some colors and is thus not setting any class index the corresponding pixels in the mask, which results in undefined values (from the empty initialization).