Hello Mr. Air!



 theairbend3r:

Mar 10 17:10:11 maverick kernel: [  319.690728] mce: CPU11: Core temperature above threshold, cpu clock throttled (total events = 75)
...
Mar 10 17:10:11 maverick kernel: [  319.690730] mce: CPU11: Package temperature above threshold, cpu clock throttled (total events = 290)



Your laptop is overheating and shutting itself down to prevent permanent
damage.
When your cpu and/or gpu is running full tilt – and pytorch tries to make
it run as fast as it can – that creates a lot of heat.  (So pytorch is doing
its job here!)


How can I stop this from happening again ie. stop pytorch training and not crash my system?


Some things you can try:
Make sure your laptop has good ventilation.  Don’t have it sitting on a
fluffy blanket or let your cat sleep blocking the cooling ports.
Make sure your cooling ports aren’t blocked or clogged up with lint.
Make sure your cooling fans are working properly.  (Lenovo made a
series of laptops where the cooling fans would fail / get noisy as they
got older.  This is something of a known issue.)
Try running your laptop on a cooling rack (those things you put your
cookies on after you take them out of the oven.)
Aim a fan or air conditioner at your laptop.
Open up the case of your laptop for better air flow – but, with a laptop,
I probably wouldn’t do this.
Squirt “compressed air” into your cooling ports – it gets quite cold as
it expands.
In your os power-management settings or bios you may have an option
to forcibly slow your laptop down.  This will slow your training down,
but it will generate less heat.
Train on the cpu instead of the gpu (or vice versa).  One or the other
may have better cooling versus heat generation.  But training on the
cpu will very likely run more slowly.
If worst comes to worst, and none of the above work, you may have to
resort to training your model in stages.  Run for a short enough amount
of time that your laptop doesn’t overheat / shut down, checkpoint your
model, let your laptop idle long enough to cool off some, reload your
checkpoint and train some more for a short period of time.  Repeat.
Good luck.
K. Frank