learning rate = 0.001
momentum = 0.5
Could this be one of this parameter fault? The change of behaviour after the first epoch looks really strange to me, almost like finetuning is going okay at the beginning, then starting from scratch in the second epoch.