Problem solved. For the sake of people who might see this post, let me summarize the problem and solution:
Problem is that floating-point operations on CPU can become very slow if numbers are “denormal” or “subnormal” 6. This means that values are very very small (smaller than 1.e-32). This is very unusual.
Workaround is to force CPU to treat these numbers as zeroes. At the beginning of your code add torch.set_flush_denormal(True) (this may not work on some older Intel CPUs though).
Alternative workaround is to “manually” remove denormals from your weights:
weight = ...
mask = (weight.abs() < 1.e-32).float()
weight = weight * mask

Better solution is to find out why your training ended up with such weird weights, and fix it!
Thanks to @albanD and others for the awesome support.