Some models such as MaskRCNN output segmentation masks as well as classification results for the objects, so you might take a look at these implementations and check, if you could use a similar workflow for your use case.