I have worked on keypoint detection before, perhaps you could try the following solution to ensure the number of keypoints are changed from 17 to the desired keypoints
model = torchvision.models.detection.keypointrcnn_resnet50_fpn(pretrained=False, progress=True, num_classes=2, pretrained_backbone=True)

model.roi_heads.keypoint_predictor = models.detection.keypoint_rcnn.KeypointRCNNPredictor(512, 2)


In my project I required the keypoint detected to be 26 as opposed to the default 17 so I changed the keypoint_predictor output in the roi_head directly.
Hope this helps