Could you rerun the code using a few seeds and check, if you get these bad results randomly using both approaches?
If the error of the manual vs. PyTorch batch norm output is small, e.g. an unlucky initialization might yield these results.