Thanks for pointing out the mistake. I think I reused my previous code for this.
I am experimenting with something which is why I have pre-defined the weights for my model. It worked for the smaller models which is why I am trying bigger models now.
So I have pre-assigned weights to my model and then I am training my model. However, as you suggested with Adam optimiser, it’s still now learning.
Epoch 0/24
----------
100%|██████████| 3125/3125 [00:12<00:00, 250.17it/s]
train Loss: 2.3923 Acc: 0.5000
100%|██████████| 3125/3125 [00:07<00:00, 402.19it/s]
val Loss: 2.3923 Acc: 0.5000

Epoch 1/24
----------
100%|██████████| 3125/3125 [00:12<00:00, 254.30it/s]
train Loss: 2.3923 Acc: 0.5000
100%|██████████| 3125/3125 [00:07<00:00, 413.95it/s]
val Loss: 2.3923 Acc: 0.5000

Epoch 2/24
----------
100%|██████████| 3125/3125 [00:12<00:00, 258.83it/s]
train Loss: 2.3923 Acc: 0.5000
100%|██████████| 3125/3125 [00:07<00:00, 417.37it/s]
val Loss: 2.3923 Acc: 0.5000

What else do you suggest?