Solved by rzp0063 in post #5 
              
              
                @ptrblck Thank you so much for your response and guidance to solve this issue with several tips. This makes a lot more sense now why the the points are shifting towards zero and confirms the source of error with this approach. 
I will keep trying to solve it using ConvNets along with your suggestion…