Solved by vahvero in post #4 
              
              
                I have found the error in the code above. The self.data is defined here 
as a List[Tensor, Dict[str, Tensor]] in the dataset class. The changes done to targets["boxes"] changed the underlying data in self.data list which meant that any epochs after the first could contain “double” changes. The boxes…