Hi all,
I come across a strange problem when I train my model on the server. At the first several iterations, the running time is OK. All the CPUs are working for IO and GPU is working. But after several iterations, suddenly the CPUs do not work for IO. I do not know what happened.
Could someone help me? Thank you very much!
During the first several iterations:

Selection_002.png1228×92 8.48 KB
 
Selection_001.png1182×463 59.1 KB

Suddenly, CPUs are not working:

Selection_004.png1247×57 5.18 KB
 
Selection_003.png1199×445 53.4 KB

And it continues like this. Very slow.