@ptrblck Thank you so much for your response and guidance to solve this issue with several tips. This makes a lot more sense now why the the points are shifting towards zero and confirms the source of error with this approach.
I will keep trying to solve it using ConvNets along with your suggestion of sorting the outputs thats a very clever idea; it would be very interesting to show that this problem could be solve with simple ConvNets approach.
I wanted to give you the good news since you have been so helpful and responsive to my question, I tried YOLO network right away over the past couple of days since it allows to have various number of objects per image. The results using the YOLO network are nearly pefect!


Again thank you so much for your help! I will keep trying with ConvNets though as it would be a more attractive solution requiring less computational power.