Hey, thanks for the reply.
You seem to have understood the issue correctly. To answer some of your questions:
I do shuffle both train sets, and I use a batch size of 512. I use a custom joint dataset which returns 512 samples of each dataset. Both datasets also use the same transform (augmentation, normalization stats, etc.). The normalization stats were originally different, but I modified my code to use the same normalization after the first occurence of this problem. Unfortunately that didnt fix anything.
Lastly, I have indeed tried training in eval() mode just to see if how that affects things. When I set eval() during the forward/backward pass of both datasets, the CE Loss shoots up to around 5000, and the train accuracy drops to 10%.
I’m not really sure how to interpret this issue. It may be a code error on my part, but I do want to mention that I use the same trainer code for smaller datasets such as mnist, svhn and cifar, and those results have been successful. The new dataset im working with contain larger images, so really the only part of my code that has changed is the dataset and the model (which is a resnet-50 model with the fc layer replaced to fit the dataset classes).