Your code might create values larger than 1. due to the limited floating point precision as seen here:
torch.manual_seed(8)

cls_score = torch.randn(2000, 20, device='cuda')
cls_score[:, 19] = 100.
det_score = torch.randn(2000, 20, device='cuda')

cls_prob = F.softmax(cls_score, dim=1) # across classes [2000,20]
det_prob = F.softmax(det_score, dim=0) # across proposals/detections [2000,20]
predict = torch.mul(cls_prob, det_prob) # shape: [2000,20]
pred_class_scores = torch.sum(predict, dim=0) # [20]
print((pred_class_scores > 1.).any())
> tensor(True, device='cuda:0')

print(pred_class_scores[19] - 1.)
> tensor(1.1921e-07, device='cuda:0')

so I think you should clamp the values before passing them to the loss function.