I am training a model on MNIST and during the training the accuracy of the model drops drastically to 10%.
Epoch: 37
468 469 Loss: 0.078 | Acc: 97.338% (58403/60000)
78 79 Loss: 0.046 | Acc: 98.860% (9886/10000)
Best Test Score tensor(98.8800)
Epoch:  38

Epoch: 38
468 469 Loss: 0.068 | Acc: 97.795% (58677/60000)
78 79 Loss: 0.053 | Acc: 98.780% (9878/10000)
Best Test Score tensor(98.8800)
Epoch:  39

Epoch: 39
468 469 Loss: 0.418 | Acc: 85.373% (51224/60000)
78 79 Loss: 2.294 | Acc: 12.560% (1356/10000)
Best Test Score tensor(98.8800)
Epoch:  40

Epoch: 40
468 469 Loss: 2.302 | Acc: 11.133% (6680/60000)
78 79 Loss: 2.301 | Acc: 10.610% (1061/10000)
Best Test Score tensor(98.8800)
=====> Saving checkpoint...
Epoch:  41

Epoch: 41
468 469 Loss: 2.304 | Acc: 11.155% (6693/60000)
78 79 Loss: 2.303 | Acc: 9.590% (959/10000)
Best Test Score tensor(98.8800)

I searched through and realized this is a bug with higher version of Pytorch.
After disabling cudnn of BN layer 3, it works if I don’t feed forward with the testset, although it slows down the training by about 1.5x time. Is there any work around to solve the problem?