Here’s my observations:

train the whole model without freezing any parameters. one epoch can finish with no problem.
freeze certain parameters. it throws “transform: failed to synchronize: cudaErrorAssert: device-side assert triggered” at random point. In my case, one epoch has 1170 batches, no shuffle, sequential feed in, it usually errors at about 600 to 800 batches.
haven’t tried on cpu, cause it takes more than 2 hours to reach ~600 batches.

This makes me think that: if not optimizing all parameters,  certain parameters may accumulate to some error status like -inf  as training goes, finally causes the error?
Any insights? Thanks.