Solution: The way I labeled the images and masks caused them to be mismatched when calling sorted(images) and sorted(masks). So the images were being labeled with the wrong mask. Fixing this gave me accurate predictions.