Fine, I figured out the reason, here is the source code from torchvision.models.detection.rpn.RPNHead:
    def forward(self, x):
        # type: (List[Tensor])
        # print([i.shape for i in x])
        logits = []
        bbox_reg = []
        for feature in x:
            t = F.relu(self.conv(feature))
            logits.append(self.cls_logits(t))
            bbox_reg.append(self.bbox_pred(t))
            print("RPN", self.cls_logits(t).shape)
            print("RPN", self.bbox_pred(t).shape)
        return logits, bbox_reg

The anchor_sizes defined above anchor_sizes = ((32,), (64,), (128,), (256,), (512,)) will be iterated by the for loop, so the RPNHead only needs rpn_anchor_generator.num_anchors_per_location()[0] as the num_anchors because all anchor_sizes share the same ratios.