I don’t think the oversampling strategy is bad at all, sorry for the misunderstanding.
Since your current issue occurs randomly, I thought that maybe some batches, which are sampled randomly at a later epoch, might have invalid or very high/low values, thus throwing your running stats off.
If you check the running stats in your training loop, could you also print the min/max/mean/std of the current batch?
Maybe it’s not related to the running stats at all, but to other parameters which might suffer from potentially bad input.
As you see, I’m currently trying to collect all your information and think about some useful debug approaches.