I have logged in to a server which has 4 NVIDIA 1080 GPUs. I ran nvidia-smi and found out that the global memory of GPU 0 is almost full but other GPUs have lot of free global memory. The status is as follows:
±----------------------------------------------------------------------------+
| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |
|-------------------------------±---------------------±---------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108…  Off  | 00000000:02:00.0 Off |                  N/A |
| 68%   87C    P2   181W / 250W |  10752MiB / 11178MiB |     98%      Default |
±------------------------------±---------------------±---------------------+
|   1  GeForce GTX 108…  Off  | 00000000:03:00.0 Off |                  N/A |
| 29%   63C    P8    22W / 250W |     25MiB / 11178MiB |      0%      Default |
±------------------------------±---------------------±---------------------+
|   2  GeForce GTX 108…  Off  | 00000000:82:00.0 Off |                  N/A |
| 20%   52C    P8    17W / 250W |     25MiB / 11178MiB |      0%      Default |
±------------------------------±---------------------±---------------------+
|   3  GeForce GTX 108…  Off  | 00000000:83:00.0 Off |                  N/A |
| 34%   67C    P2    63W / 250W |    536MiB / 11178MiB |      0%      Default |
±------------------------------±---------------------±---------------------+
I am trying to run the code snippet where the CNN is shallow with 3 convolution layers:
device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
cnn = CNN()
cnn = cnn.to(device)
Its clear that I want to run this on CUDA device 1 which has got ample global memory. But when I run it I am getting error in the line: cnn = cnn.to(device) as:
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCTensorRandom.cu:25
Why is this so? Can somebody help me? Thanks in advance.
Some details:
OS: Ubuntu server 16.04
pytorch version: 0.4.0
python version: 3.5
package manager: pip
CUDA version: 8.0.61
CUDNN version: 7.1.02