Hey all,
I’m running some models from the model-zoo; using these examples: https://github.com/pytorch/examples/tree/master/imagenet
Given for example alexnet, I try to retrain it, but the performance tanks dramatically. Using just this cloned repo, running: -a alexnet --lr 0.01 --pretrained [image_net location]
performance starts off strong, but in a few iterations already decreases from 80ish% accuracy to 63 and stays there. First few epochs:
Epoch: [0][0/5005]	Time 8.089 (8.089)	Data 3.601 (3.601)	Loss 2.0473 (2.0473)	Prec@1 54.297 (54.297)	Prec@5 78.906 (78.906)
Epoch: [0][10/5005]	Time 0.856 (1.183)	Data 0.787 (0.611)	Loss 2.4228 (2.2277)	Prec@1 49.219 (50.639)	Prec@5 71.484 (73.864)
Epoch: [0][20/5005]	Time 1.752 (1.089)	Data 1.675 (0.747)	Loss 2.3623 (2.3440)	Prec@1 47.266 (48.400)	Prec@5 71.484 (71.540)
Epoch: [0][30/5005]	Time 0.175 (1.029)	Data 0.000 (0.773)	Loss 2.9376 (2.4701)	Prec@1 36.328 (45.867)	Prec@5 62.891 (69.619)
Epoch: [0][40/5005]	Time 1.708 (1.010)	Data 1.632 (0.797)	Loss 2.8086 (2.5262)	Prec@1 39.453 (44.741)	Prec@5 66.016 (68.807)
Epoch: [0][50/5005]	Time 1.624 (0.993)	Data 1.543 (0.805)	Loss 2.6798 (2.5577)	Prec@1 42.969 (44.118)	Prec@5 62.109 (68.252)
Epoch: [0][60/5005]	Time 0.312 (0.984)	Data 0.249 (0.811)	Loss 2.4672 (2.5890)	Prec@1 47.266 (43.654)	Prec@5 70.312 (67.802)
Epoch: [0][70/5005]	Time 1.153 (0.974)	Data 1.082 (0.812)	Loss 2.5792 (2.6097)	Prec@1 44.141 (43.277)	Prec@5 70.703 (67.463)
Epoch: [0][80/5005]	Time 0.200 (0.979)	Data 0.018 (0.823)	Loss 2.5768 (2.6335)	Prec@1 44.141 (42.670)	Prec@5 68.359 (67.086)
Things I tried:

Make sure the dataset is correct, it is on comparison with someone else’s instance
retrain alexnet from scratch; currently on epoch 14 and Prec@5 is around 58 now. Lower than the 63ish from the retraining network, but I guess this should go up for the next 90 epochs (taking a while of course).
Changing learning rates. Lower learning rates cause more stable learning of course, but unless I put the learning rate on 1e-5 performance still drops a lot.
Changing optimization method; Adam seems to decrease results even more dramatically in the first few iterations.
I also tried this for inception v3 (my initial goal), which has the same problems.
Interestingly, test scores get higher than the train scores. on epoch 12 of the retraining, the net scores around * Prec@1 47.254 Prec@5 72.312 . on testing data, as opposed to Prec@1 37.891 (38.853)	Prec@5 64.062 (63.824) on the training data. That seems weird to me.

Anyone have an idea of what I’m missing here? I reckoned the retraining/finetuning should just work out of the box. Perhaps there’s some small difference when doing things like data-preprocessing in this code, but then I wouldn’t think performance would drop hard in the first iterations…
Specs: Python Python 3.5.2. pytorch version ‘0.2.0_3’.