Object classes should be labelled sequentially, starting from 1. So, if you have 4 object classes, they should be numbered 1, 2, 3, and 4 in the labels variable mentioned by OP.
However, when creating the FasterRCNN object, the documentation for the num_classes argument says that you have to account for the background, which, by default, is class number 0.

  

      github.com
  

  
    pytorch/vision/blob/850491eb9c3daff887468718bfa1f90f94042d9d/torchvision/models/detection/faster_rcnn.py#L54-L55 2



      
      
          num_classes (int): number of output classes of the model (including the background).
              If box_predictor is specified, num_classes should be None.
      
    
  


  

  
    
    
  

  


So, in our example with 4 object classes, we actually have to pass num_classes=5 to FasterRCNN, even though we don’t have 4 classes of object annotations in our dataset.