How large is your batch size during fine-tuning? Note that the updates for the running stats might be off, if the batch is too small. You could change the momentum to smooth the updates a bit in this case.
Also, does the accuracy recover at some point?


During evaulation the running estimates of mean and std are used instead of the current batch statistics.