When you apply your horizontal flipping, you most likely aren’t re-assigning keypoints, so the network is learning to get the best of both worlds and output the average. Namely, you need not only only flip the values of the keypoints, but also re-map them to their new keypoints, i.e. the left eye points now go to the right eye points.