That’s an interesting use case.
If I understand your current approach correctly, you are using the first output indices for the valid points and set the rest to the zero target.
Assuming that’s the case, then this would explain the “drift” of your points towards the zero location.
Some of the “later” outputs will get zero targets more often than the earlier ones and the model could try to output the “mean” location, which would be somewhere between a valid location and the zero point.
I’m not sure what the best approach would be, but I guess you could try to reuse some approaches of object detection models, e.g. sorting the outputs based on their mse loss to the target value (in case your model just predicts the points in a different order) or work with proposals and keep the best ones.