I encountered with similar problem after freezing BN. Yet my loss doesn’t raise to a very high magnitude, but just shifts to NaN abruptly at some epoch. Anyway, the problem seems gone after I tuning down the learning rate, but I still don’t know the cause of it.