I’m using jupyter. After the crash, I cannot ssh it. I have to restart the server by power. It seems like a serious problem.
In my opinion, the crash may be related to three reasons.

interrupt during the training
data parallel
delete model and empty cache

But anyway my goal is not this. I just want to make sure my results reproducible. Could you give me some advices? I will explain the problem briefly.
I want to test 4 BERT models. Let’s call them ABCD. (I will restart kernel after training. )
If I train A only , the result is always the same.
If I train ABCD by a for loop, the result for ABCD is also always the same.
Here the problem comes. If I train B only, this result is different from the result of B trained in a for loop.
In my opinion, the result of training B only is correct since  it doesn’t get any effect from other things. I am confused about how A can affect B in the training loop. From the code, you can see that I initialize the model and optimizer in each loop.