Thanks a lot for the guidance. So after recreating the optimizer, training is happening:
Entire code here: https://pastebin.com/kjxu9fHB 47 I added two lines here:
for idx in range(nb_samples):
 
    print('Using sample {} as test data'.format(idx))
 
    print('Resetting model')
    model_ft = copy.deepcopy(initial_model)
    # Observe that all parameters are being optimized
    optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)
 
    # Decay LR by a factor of 0.1 every 7 epochs
    exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
 
    # Get all indices and remove test sample
    train_indices = list(range(len(image_datasets['train'])))
    del train_indices[idx]

Using sample 49 as test data
Resetting model
Epoch 0/24
----------
Loss: 1.0131 Acc: 0.5400
Epoch 1/24
----------
Loss: 1.2152 Acc: 0.4400
Epoch 2/24
----------
Loss: 1.1712 Acc: 0.5200
Epoch 3/24
----------
Loss: 2.2373 Acc: 0.5200
Epoch 4/24
----------
Loss: 0.8226 Acc: 0.6800
Epoch 5/24
----------
Loss: 0.9330 Acc: 0.4600
Epoch 6/24
----------
Loss: 0.6816 Acc: 0.6200
Epoch 7/24
----------
Loss: 0.2960 Acc: 0.8800
Epoch 8/24
----------
Loss: 0.2573 Acc: 0.9200
Epoch 9/24
----------
Loss: 0.2766 Acc: 0.9400
Epoch 10/24
----------
Loss: 0.2305 Acc: 0.9000
Epoch 11/24
----------
Loss: 0.1655 Acc: 0.9400
Epoch 12/24
----------
Loss: 0.1975 Acc: 0.9400
Epoch 13/24
----------
Loss: 0.2006 Acc: 0.9400
Epoch 14/24
----------
Loss: 0.1893 Acc: 0.9400
Epoch 15/24
----------
Loss: 0.1941 Acc: 0.9400
Epoch 16/24
----------
Loss: 0.1841 Acc: 0.9600
Epoch 17/24
----------
Loss: 0.2250 Acc: 0.9000
Epoch 18/24
----------
Loss: 0.2015 Acc: 0.9400
Epoch 19/24
----------
Loss: 0.1792 Acc: 0.9200
Epoch 20/24
----------
Loss: 0.1708 Acc: 0.9800
Epoch 21/24
----------
Loss: 0.1901 Acc: 0.9200
Epoch 22/24
----------
Loss: 0.1701 Acc: 0.9800
Epoch 23/24
----------
Loss: 0.1362 Acc: 0.9800
Epoch 24/24
----------
Loss: 0.1623 Acc: 0.9400
Training complete in 1m 7s
<class 'torch.Tensor'>
torch.Size([1, 3, 224, 224])
loocv preds:  [tensor([1], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([1], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([1], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([1], device='cuda:0'), tensor([1], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([1], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0'), tensor([0], device='cuda:0')]
loocv targets:  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
acc score:  0.54
confusion matrix: 
 [[23  2]
 [21  4]]


Also, for
print(‘Class probabilities: {}’.format(F.softmax(output, 1)))
I get:
Class probabilities: tensor([[0.9514, 0.0486]], device=‘cuda:0’, grad_fn=)
Which only shows 2 numbers, not like 50 (images) * 2 (classes) numbers.
Questions:

Should I pass the entire dataloader to train_model?
How can I use F.softmax(output, 1)) to show 50*2 numbers for probabilities.

Thanks a lot.