Based on the description it seems that your overall training is unstable and the success rate of the training could be low. You could retrain N models using different seeds for each run and check how many times the model converges properly vs. a failure. To stabilize it, you could try to use other parameter initialization methods etc.



 Rohit_R:

and think it might be due to PyTorch’s random seed or with cuda memory allocation.


I don’t quite understand how the CUDA memory allocation could be related to this issue. Could you explain your concern a bit more?