Hi damonbla,
Faster RCNN from torchvision is built upon several submodels and two of them are trained in the process:
-A RPN for computing proposal regions (computes absence or presence of classes + region proposals)
-A FasterRCNN Predictor (computes object classes + box coordinates).
These submodels are already implementing the loss function that you can find in the associated papers and therefore, you don’t need to bother.
More, it appears that you cannot use your own loss function with the current torchvision implementation.
To understand how it is trained, I encourage you to read the function “train_one_epoch” on github which is used in the tutorial to train the fasterrcnn_resnet50_fpn:

  
      github.com
  
  
    pytorch/vision/blob/master/references/detection/engine.py 226
import math
import sys
import time
import torch

import torchvision.models.detection.mask_rcnn

from coco_utils import get_coco_api_from_dataset
from coco_eval import CocoEvaluator
import utils


def train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq):
    model.train()
    metric_logger = utils.MetricLogger(delimiter="  ")
    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
    header = 'Epoch: [{}]'.format(epoch)

    lr_scheduler = None
    if epoch == 0:


  This file has been truncated. show original

  
  
    
    
  
  


I am new to pytorch so I may be wrong, so if you found out something different don’t hesitate to correct me.
Cheers,
Tristan