The issue might arise from the changed norm layers, but I haven’t seen a similar issue before.
You could try to isolate the problematic sample and check the output as well as the internal stats of the norm layers to check, if these layers are responsible for the drop in accuracy.