Hello @ptrblck thank you for your reply. I added more images. Now my training data size is 1648 + 1635. Here are the Augmentations I used:
batch_size = 32
Size of validation set = 128
train_transform = torchvision.transforms.Compose([

    torchvision.transforms.Resize(size=258),

    torchvision.transforms.RandomHorizontalFlip(),

    torchvision.transforms.RandomResizedCrop(224),

    #torchvision.transforms.RandomVerticalFlip(p=0.5),

    torchvision.transforms.RandomRotation(10),

    torchvision.transforms.ToTensor(),

    torchvision.transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.25, 0.25, 0.25])

])

val_transform = torchvision.transforms.Compose([

    torchvision.transforms.Resize(size=258),

    torchvision.transforms.RandomResizedCrop(224),

    #torchvision.transforms.RandomHorizontalFlip(),

    #torchvision.transforms.RandomVerticalFlip(p=0.5),

    #torchvision.transforms.RandomRotation(2.8),

    torchvision.transforms.ToTensor(),

    torchvision.transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.25, 0.25, 0.25])

])

Here are the training codes:
transfer_model = models.resnet50(pretrained=True)

for name, param in transfer_model.named_parameters():

    if("bn" not in name):

        param.requires_grad = False

transfer_model.fc = nn.Sequential(nn.Linear(transfer_model.fc.in_features,500),

nn.ReLU(),                                 

nn.Dropout(), nn.Linear(500,2))

if torch.cuda.is_available():

    device = torch.device("cuda") 

else:

    device = torch.device("cpu")

transfer_model.to(device)

optimizer = optim.SGD(transfer_model.parameters(), lr=0.001, momentum=0.9)

Statisitics:
Epoch: 0, Training Loss: 0.68, Validation Loss: 0.67, accuracy = 0.55
Epoch: 1, Training Loss: 0.66, Validation Loss: 0.63, accuracy = 0.68
Epoch: 2, Training Loss: 0.62, Validation Loss: 0.55, accuracy = 0.77
Epoch: 3, Training Loss: 0.55, Validation Loss: 0.51, accuracy = 0.70
Epoch: 4, Training Loss: 0.48, Validation Loss: 0.39, accuracy = 0.83
Epoch: 5, Training Loss: 0.41, Validation Loss: 0.35, accuracy = 0.85
Epoch: 6, Training Loss: 0.38, Validation Loss: 0.35, accuracy = 0.83
Epoch: 7, Training Loss: 0.33, Validation Loss: 0.38, accuracy = 0.80
Epoch: 8, Training Loss: 0.30, Validation Loss: 0.25, accuracy = 0.91
Epoch: 9, Training Loss: 0.29, Validation Loss: 0.28, accuracy = 0.88
Epoch: 10, Training Loss: 0.27, Validation Loss: 0.25, accuracy = 0.88
Epoch: 11, Training Loss: 0.24, Validation Loss: 0.16, accuracy = 0.91
Epoch: 12, Training Loss: 0.24, Validation Loss: 0.16, accuracy = 0.94
Epoch: 13, Training Loss: 0.24, Validation Loss: 0.19, accuracy = 0.91
Epoch: 14, Training Loss: 0.23, Validation Loss: 0.22, accuracy = 0.91
Epoch: 15, Training Loss: 0.22, Validation Loss: 0.15, accuracy = 0.93
Epoch: 16, Training Loss: 0.21, Validation Loss: 0.22, accuracy = 0.84
Epoch: 17, Training Loss: 0.21, Validation Loss: 0.15, accuracy = 0.92
Epoch: 18, Training Loss: 0.19, Validation Loss: 0.16, accuracy = 0.94
Epoch: 19, Training Loss: 0.20, Validation Loss: 0.13, accuracy = 0.93

How may I get a good result?