ericlormul:

if not optimizing all parameters, certain parameters may accumulate to some error status like -inf as training goes, finally causes the error?


This could be the case and the assertion could point towards e.g. a failing index operation.
Could you rerun the code via:
CUDA_LAUNCH_BLOCKING=1 python script.py args

and post the complete stack trace here?