Hi Simon, thanks a lot for the reply, I totally agree with you, but first, I am trying to reproduce the formal imagenet tutorial: https://github.com/pytorch/examples/blob/master/imagenet/main.py 41
Second, when I changed the iteration part in the train_model function as follows per your feedback:
    # Iterate over data.
    for i, (inputs, target) in enumerate(dataloaders['train']):

        target = target.cuda(async=True)
        inputs = inputs.cuda(async=True)
        input_var = torch.autograd.Variable(inputs)
        target_var = torch.autograd.Variable(target)

        # compute output
        output = model(input_var)
        loss = criterion(output, target_var)

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Now, I get a new error:

Traceback (most recent call last):
File “finetunacuda.py”, line 299, in 
main()
File “finetunacuda.py”, line 297, in main
fine_tuna_protocol()
File “finetunacuda.py”, line 275, in fine_tuna_protocol
model_ft = train_model(dataloaders, dataset_sizes, model_ft, criterion, optimizer_ft, num_epochs = nep, temp_save_name = name_of_results_output_txt_file)
File “finetunacuda.py”, line 215, in train_model
output = model(input_var)
File “/home/ubuntu/envs/deepL/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 325, in call
result = self.forward(*input, **kwargs)
File “/home/ubuntu/envs/deepL/lib/python3.5/site-packages/torchvision/models/resnet.py”, line 142, in forward
x = self.maxpool(x)
File “/home/ubuntu/envs/deepL/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 325, in call
result = self.forward(*input, **kwargs)
File “/home/ubuntu/envs/deepL/lib/python3.5/site-packages/torch/nn/modules/pooling.py”, line 143, in forward
self.return_indices)
File “/home/ubuntu/envs/deepL/lib/python3.5/site-packages/torch/nn/functional.py”, line 334, in max_pool2d
ret = torch._C._nn.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58

What do you think? I run this on a p2.xlarge EC2 instance on AWS.
Thanks a lot for the assistance
Hassan