Issue Description
I tried to train my model on multiple gpus. However, when I launch the program, it hangs in the first iteration. Using nvidia-smi, i find hundreds of MB of memory is consumed on each gpu. I guess these memory usage is for model initialization in each gpu.
I am sharing 8 gpus with others on the server, so I limit my program on GPU 2 and GPU 3 by following command.
os.environ['CUDA_VISIBLE_DEVICES'] = '2,3'

Code example
I ran this official tutorial on my machine and the same thing happens again. https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html 89
System Info
PyTorch version: 0.4.0
Is debug build: No
CUDA used to build PyTorch: 8.0.61
OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
CMake version: version 3.5.1
Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 7.5.17
GPU models and configuration:
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X
GPU 2: GeForce GTX TITAN X
GPU 3: GeForce GTX TITAN X
GPU 4: GeForce GTX TITAN X
GPU 5: GeForce GTX TITAN X
GPU 6: GeForce GTX TITAN X
GPU 7: GeForce GTX TITAN X
Nvidia driver version: 375.88
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.5.1.10
/usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.20
/usr/lib/x86_64-linux-gnu/libcudnn_static_v5.a
/usr/lib/x86_64-linux-gnu/libcudnn_static_v6.a
/usr/local/lib/python2.7/dist-packages/torch/lib/libcudnn.so.6
Versions of relevant libraries:
[pip3] msgpack-numpy (0.4.1)
[pip3] numpy (1.14.0)
[pip3] numpydoc (0.7.0)
[pip3] torch (0.4.0)
[pip3] torchtext (0.2.3)
[pip3] torchvision (0.2.1)
[conda] torch                     0.4.0                     
[conda] torchtext                 0.2.3                     
[conda] torchvision               0.2.1                     
Solution for those who face similar problem

  
      github.com/pytorch/pytorch
  
  
    



Issue: Multi-GPU K80s 53


	
	opened by davidmascharka
	on 2017-05-23
	
	
	closed by apaszke
	on 2017-10-22
	


I'm having trouble getting multi-gpu via DataParallel across two Tesla K80 GPUs. The code I'm using is a modification of the...