It’s hard to understand the situation in detail without any code; are you referring to test accuracy that is run on a separate test dataset?
It might be useful to verify that the test accuracy is consistent when saving a checkpoint vs. when loading the same checkpoint (without any additional training).