Thank you @jerryzh168. So, I tried to quantize the mobilenetv3 backbone only. I modified the mobilenet_backbone() function to obtain the quantized version of the backbone as follows:
#backbone = mobilenet.__dict__[backbone_name](pretrained=pretrained, norm_layer=norm_layer).features
backbone = torchvision.models.quantization.mobilenet_v3_large(pretrained=False, quantize=False).features

Then, I followed this tutorial 3 to create and train a FasterRCNN model. Before training, I fused the model backbone as follows:
# Create quantized model
fused_model = get_quantized_object_detection_model(num_classes, pretrained=True)
fused_model.to(cpu_device)
fused_model.train()
# Fuse layers
for m in fused_model.backbone.modules():
    if type(m) == ConvBNActivation:
        modules_to_fuse = ['0', '1']
        if type(m[2]) == nn.ReLU:
            modules_to_fuse.append('2')
        fuse_modules(m, modules_to_fuse, inplace=True)
    elif type(m) == QuantizableSqueezeExcitation:
        fuse_modules(m, ['fc1', 'relu'], inplace=True)
    elif type(m) == QuantizableInvertedResidual:
        for idx in range(len(m.block)):
            if type(m.block[idx]) == nn.Conv2d:
                fuse_modules(m.block, [str(idx), str(idx + 1)], inplace=True)

Then, prepared the model for quantization aware training:
backend = 'fbgemm'
fused_model.qconfig = torch.quantization.get_default_qat_qconfig(backend)
torch.quantization.prepare_qat(fused_model.backbone, inplace=True)
fused_model.to(cuda_device)

    # ... creating optimizer and training

fused_model.to(cpu_device)
torch.quantization.convert(fused_model.backbone, inplace=True)
fused_model.eval()

    # ... save model

During preparation I did not get any error, but after quantization the model size did not decrease (74MB) and inference speed and accuracy decreased. Obviously something is not right. Any thoughts?