Although I haven’t been able to completely debug this issue yet, I have found a culprit of the overall issue. So in case anyone runs into a similar issue, here is my final response:


I ran multiple tests in which I sequentially stripped the training procedure of one forward pass, until the point where I was only forward passing a single batch from a single dataset. It turns out one major issue (?) was that my pre-trained model was pre-trained with SGD momentum, while the main trainer was using Adam (with the same learning rate). For some reason, this was causing my train accuracy to plummet from 96% to 10% in the first epoch.


I tried a variety of combinations with batch norm. It seems like freezing the batch norm running stats by changing the BN layers to eval mode yields the most stable results.


While I haven’t been able to implement this yet, I think the best way to use batch norm with drastically different datasets is to keep a separate running mean/std for each dataset. While it is a little burdensome, I believe this is the most stable way of using batch norm in this case.