Hi everybody,
After going no-where with trying to set up cuda and cuddnn on my own, my mentor recommended I use the aws deep learning ami, which I did. This instance comes with preconfigured and installed python, pytorch, cuda, cuddNN.


I followed the steps in this tutorial to launch the instance: https://aws.amazon.com/blogs/ai/get-started-with-deep-learning-using-the-aws-deep-learning-ami/ 12


After starting the instance I confirmed everything is properly installed as follows:



source activate pytorch_p27
python
import torch
torch.version
’0.3.0.post4’
torch.cuda.is_available()
True
torch.backends.cudnn.version()
7003

I am still getting the runtime error out of memory trying to finetune resnet18 with a batchsize of 1 and I use images with standard size of 3x224x224:

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1512378422383/work/torch/lib/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File “finetunacuda.py”, line 299, in 
main()
File “finetunacuda.py”, line 297, in main
fine_tuna_protocol()
File “finetunacuda.py”, line 275, in fine_tuna_protocol
model_ft = train_model(dataloaders, dataset_sizes, model_ft, criterion, optimizer_ft, num_epochs = nep, temp_save_name = name_of_results_output_txt_file)
File “finetunacuda.py”, line 215, in train_model
output = model(input_var)
File “/home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 325, in call
result = self.forward(*input, **kwargs)
File “build/bdist.linux-x86_64/egg/torchvision/models/resnet.py”, line 142, in forward
File “/home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 325, in call
result = self.forward(*input, **kwargs)
File “/home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/nn/modules/pooling.py”, line 143, in forward
self.return_indices)
File “/home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/nn/functional.py”, line 334, in max_pool2d
ret = torch._C._nn.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1512378422383/work/torch/lib/THC/generic/THCStorage.cu:58

I did everything I got my hands on, right now, I have no idea why I can’t get to run.
Here is the output of nvidia-smi
Wed Jan  3 03:58:23 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.81                 Driver Version: 384.81                    |
|-------------------------------±---------------------±---------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:00:1E.0 Off |                    0 |
| N/A   44C    P8    30W / 149W |      1MiB / 11439MiB |      0%      Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
±----------------------------------------------------------------------------+
Any feedback is greatly appreciated. I am posting this issue here, in case I couldn’t get any feedback I will have to start a new post.
Thanks