I already solved this problem code wise, but I don’t know why what I did worked and want an explanation.
I was doing SSD bounding box detection and loss was okay in the first epoch. Then in the second epoch, it suddenly jumped to 5000-10000 range. This loss jump was coming from my loss function for the location of bounding box, meaning x y coordinates of where the object is; loss function for labeling, classifying the class of the object being detected was okay, going down gradually as expected
I figured out that if I don’t do data augmentations like random flip, random crop, random hue change and stuff of those sorts in my data loader, the loss for xy coordinates doesn’t go crazy. Then I set the random seed and enabled data augmentation, and it also worked fine and loss was stable.
torch.manual_seed(1234)
np.random.seed(1234)
random.seed(1234)

Could anyone explain to me what is going on?