Could you post the scale of the loss as well?
Also, how reproducible is this issue? I.e. if you rerun the code with different seeds is the run on the 3090 always diverging earlier? Since the model is diverging in each case, I’m unsure if an “earlier divergence” would be problematic (if would be, if the model is always converging in one setup).