Thanks Alex , yes it doesn't work. So,this time around I saved the model state_dict and also the full model.

As I started training again, I had to stop in the middle because of power cut. I just wanted to resume training.
This time around, the load_state_dict still had problems with the size but loading the full model I was able to resume training with the current loss.

Here it is, when I started the GAP score was 0.0000
1. epoch 1
total batches: 5024
1 [0/5024]      time 237.078 (237.078)  loss 9.9302 (9.9302)    GAP 0.0000 (0.0000)

After resuming, loading the full model , the GAP score is below.
2. epoch 1
total batches: 5024
1 [0/5024]      time 68.160 (68.160)    loss 5.0118 (5.0118)    GAP 0.0345 (0.0345)

So, the solution to loading state_dict when using pretrained model is to avoid copying fc.weights and fc.bias

avoid = ['fc.weight', 'fc.bias']
for key  in pytorch_state_dict.keys():
        if key in avoid or key not in state_dict.keys(): # 
            # print('not in', key)
            continue
        if pytorch_state_dict[key].size() != state_dict[key].size():
            # print('size not the same', key)
            continue
        state_dict[key] = pytorch_state_dict[key]

Thanks to Arunava Chakraborty also for his advice