The loss or accuracy calculation might be wrong, e.g. you would accumulate the loss for various epochs or use unwanted broadcasting in the accuracy calculation.
I’m just speculating, as it’s hard to debug without seeing some code, so feel free to add more information or code.