Solved by ptrblck in post #2 
              
              
                The classification models were pretrained on ImageNet, while the segmentation and detection models use e.g. COCO. The pretrained argument gives you information about the used dataset.