Prinsphield:

even though I turn off the display


Not sure, but does that really turn off the GPU use (you can check via Nvidia smi, RAM use should be ~0)? I would just try to simply unplug the cable for the graphics card temporarily.

motherboard, cpu and other components.

Motherboard:
Supermicro X11DPG-OT-CPU, https://www.supermicro.com/products/motherboard/Xeon/C620/X11DPG-OT-CPU.cfm 1
CPUs:
4 8-core Intel® Xeon® Silver 4110 CPU @ 2.10GHz (https://ark.intel.com/content/www/us/en/ark/products/123547/intel-xeon-silver-4110-processor-11m-cache-2-10-ghz.html)
Memory:
376GiB

how you installed pytorch and cuda?

I am using conda – I noticed it’s much faster than the custom-compiled version. Probably because of mmkl in conda since I have intel CPUs.
I.e., I am just using
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch