Hi ptrblck,
I hope you are well.
I have looked at the output from the last 3 epochs before failing and the only thing that looks strange is in the epoch before NaN appears a single value in the mean tensor goes to NaN, though there should be no reason for this.
chaslie
ps. this is related to my other post about cuda run time, which i am currently looking into.