I have a trained CNN with 70% accuracy, I apply transfer learning, add Global Average Pooling to it, and finetune this new model. However, the train & validation accuracy differ wildly depending on PyTorch’s random weight initialisation values (I’ve seen from 10% to 50% so far), even across different learning rates (1e-01 to 1e-06). The train & valid loss are simply stuck at around their initial value, even after 400 epochs of backpropagation.
I have made sure to set the model’s parameters with requires_grad() = True. What could be a possible reason for this bizarre behaviour? Thank you!