Thanks for the update. Based on the printed stats it seems you are overtraining your model, as the accuracy is noisy around 97.2%, so the updates might knock the model out of the “good” parameter space.
You could try to lower the learning rate of your optimizer(s) or use an adaptive optimizer such as Adam.
Also, were you able to reproduce this effect without using cudnn?