This issue seems to be quite weird, as these lines of code 2 shouldn’t create out of bounds indices:
# Removing some code for better readability
positive = torch.nonzero(matched_idxs_per_image >= 1).squeeze(1)

num_pos = int(self.batch_size_per_image * self.positive_fraction)
# protect against not enough positive examples
num_pos = min(positive.numel(), num_pos)

# randomly select positive and negative examples
perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]

pos_idx_per_image = positive[perm1]

It seems the last line of code is creating this issue.
My best guess is that the shape of some tensors are not what is expected.
Did you change anything in the model implementation or could you explain your use case a bit?
EDIT: Apparently I’m too late.