Do you mean the training accuracy (resubstitution) or the validation accuracy?
It’s common to see the training accuracy dropping a bit, since the model is “smaller”, i.e. is should have less capacity. The validation accuracy however shouldn’t go down.
Have you checked to use model.train() and model.eval() for training and evaluation respectively?