single gpu works fine. Dgx machine works fine. I can’t see a pattern on which gpu is crashing on me.
I don’t use DataParallel so no.
Yea I know it’s suboptimal but sometimes due to the laws of diminishing returns the last tiny gain (which is just that my script doesn’t print an errort) isn’t worth the (already days/weeks of effort) I put into solving it. It’s also not really my job to solve pytorch not work. So I am happy with my current solution as long as it trains to completion as it is now