I have found the error in the code above. The self.data is defined here
as a List[Tensor, Dict[str, Tensor]] in the dataset class. The changes done to targets["boxes"] changed the underlying data in self.data list which meant that any epochs after the first could contain “double” changes. The boxes would be changed twice and be incorrect in the subsequent epochs.
I changed the code to
    def __getitem__(self, index: int) -> Tuple[torch.FloatTensor, dict]:
        """
            Returns:
                source (tensor): Tensorof size self.__size
                target (dict): Dictionary with keys
                - boxes (FloatTensor[N, 4]): The ground-truth boxes
                 [x1, y1, x2, y2]
                - labels (Int64Tensor[N]): The class label of each image
        """
        image, targets = self.data[index]
        # Force copies
        image = image.clone().detach()
        targets = dict(targets)
        if self.transform:
            _, xmax, ymax = image.shape
            device = targets["boxes"].device
            # Horizontal flip
            if random.random() > .5:
                image = tf.vflip(image)
                # Change boxes coordinates
                targets["boxes"] = torch.abs(
                    torch.tensor(
                        [0, ymax, 0, ymax],
                        device=device
                    ) - targets["boxes"]
                )
                # Make y1 < y2
                targets["boxes"][:, [1, 3]] = targets["boxes"][:, [3, 1]]

            # Vertical flip
            if random.random() > .5:
                image = tf.hflip(image)
                targets["boxes"] = torch.abs(
                    torch.tensor([xmax, 0, xmax, 0], device=device) - targets["boxes"]
                )
                # Change the columns to make x1 < x2
                targets["boxes"][:, [0, 2]] = targets["boxes"][:, [2, 0]]

            # Color Jitter
            if random.random() > .5:
                image = tv.transforms.ColorJitter()(image)

        return image, targets

This seems to have fixed the issue. I am still running the training. I came back to this for completely unrelated reasons.