QQ截图20170321130328.jpg841×129 15.1 KB

firstly I’ve training on a regular python file using 4 GPUs using function Dataparallel.
Then I loaded saved parameters using ipython notebook through SSH while the previous job is still running.
when I load it on a single GPU instead of Dataparallel, it shows that weight doesn’t exist.So instead I use Dataparallel function on the same GPUs just like training process, then the problem occurred.
The ipython froze, and I immediately kill the job. Then My GPU is locked up like the picture shows. I can restart any job but it only shows 1MB memory whatever I tried.
I’ve ran into the same problem before, reboot can do but my peers are also using this remote server. What can I do ;-(, I searched through ‘ps -ef’ but still cannot find relevant jobs that caused the problem.
What have I done ;-(.