Maybe your model might just have learned the “mean prediction value” and thus would need some more training.
Could you check the bias values of the output layer and compare them to the output values?
I’ve seen some cases where my model get stuck during training and just predicted the mean of all target values (facial key points).