actually ran the code with Python script and get the same CUDA OUT OF MEMORY error.
----------
Loss: 0.1821 Acc: 0.9200
Training complete in 0m 37s
<class 'torch.Tensor'>
torch.Size([1, 3, 224, 224])
Using sample 34 as test data
Resetting model
Epoch 0/24
----------
Exception ignored in: <bound method _DataLoaderIter.__del__ of <torch.utils.data.dataloader._DataLoaderIter object at 0x7ff5fc18cf60>>
Traceback (most recent call last):
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 399, in __del__
    self._shutdown_workers()
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 378, in _shutdown_workers
    self.worker_result_queue.get()
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/multiprocessing/queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 151, in rebuild_storage_fd
    fd = df.detach()
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/multiprocessing/reduction.py", line 153, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "new_loocv.py", line 202, in <module>
    model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, dataloader, num_epochs=25)
  File "new_loocv.py", line 99, in train_model
    optimizer.step()
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/optim/sgd.py", line 97, in step
    buf = param_state['momentum_buffer'] = torch.zeros_like(p.data)
RuntimeError: CUDA error: out of memory

I am not sure if this answer directly related to this situation RuntimeError: cuda runtime error (2) : out of memory @ptrblck but how can we send data in batches of say size 7 or 10 or constant K in this scenario? I mean if the problem is related to batch size.
A note that I ran the same code last night and it was working. So I am not exactly sure why it is running out of cuda memory now. As you see below, output of $nvtop, I am only using 11% of the memory when running the code:

Screenshot from 2018-11-26 20-45-16.png1665×240 55.9 KB

Edit: I figured what caused the problem: appending to the loocv_probs. After commenting it, I was able to do the training.
    print("probs: ", F.softmax(output, 1))
    #loocv_probs.append(F.softmax(output, 1))

Edit2:
The following fixed it:
    gpu_tensor_probs = F.softmax(output, 1)
    cpu_numpy_probs = gpu_tensor_probs.data.cpu().numpy()
    loocv_probs.append(cpu_numpy_probs)