Hi Mr. Air!



 theairbend3r:

So, it’s all because of overheating.


Almost certainly.  Your log file was complaining about your CPUs
overheating right before the shutdown.


GPU memory or Power input has no role to play here?


I would think that you would get an explicit GPU out-of-memory
error if GPU memory were the problem.  Certainly not a hard
system shutdown.
There are power supplies (and batteries) that will shut themselves
down abruptly if they start to overheat, but, again, your log did say
that your CPUs were getting hot.


Is there anything else I can do from PyTorch to alleviate this?


You could try running on just the CPU.  (That may or may not help,
depending on the detailed cause of the overheating.)
You could also put time.sleep (5) calls in your code – perhaps in
the loop where you loop over training batches – to give your machine
a chance to cool off a little.  (Adjust the 5 by experimentation to find
how much you need to sleep to not overheat.)  Of course, this will
slow your training down.
Best.
K. Frank