sunfishcc:

65 Acc: 59.12%, Test Loss: 0.6608 Acc: 65.80%
 2/100[0:00:00.762435] Train Loss: 0.5176 Acc: 74.36%, Test Loss: 0.5115 Acc: 70.30%
 3/100[0:00:00.720053] Train Loss: 0.4273 Acc: 80.51%, Test Loss: 0.3375 Acc: 89.10%
 4/100[0:00:00.732783] Train Loss: 0.4059 Acc: 82.93%, Test Loss: 0.8257 Acc: 49.05%
 5/100[0:00:00.762209] Train Loss: 0.4373 Acc: 78.19%, Test Loss: 0.1853 Acc: 97.50%
 6/100[0:00:00.745140] Train Loss: 0.4131



Training loss should not increase during training. Could you decrease the learning rate? I think that overshooting is happening, especially between epochs 18 and 19. I would also suggest using Adam optimizer instead of SGD. It will converge faster.