I’ve faced same problem. But, on my server, this problem is caused by that there is no enough memory on GPU devices the program is using. You may specify another GPU for your program by using torch.cuda.set_device(id_of_idle_device).
Hop this can help you.