Can you confirm that your target["masks"] look like:
dtype:  torch.uint8 
shape: torch.Size([2, 480, 320])
min: tensor(0, dtype=torch.uint8) 
max: tensor(1, dtype=torch.uint8)

Where 2 is the number of objects (instance segmentation), 480 and 320 are the height and width of the input image respectively.
If, for some reason, the dimensions in the masks shape are permuted, this behavior arises.