So, I just read this post 16 here from 4 odd years ago, and may be I’m asking too much.
Maybe a better approach as the post suggests to be split the image into grids and ask it to give me the probability it is that grid?    It won’t be as precise, but then again I suppose the final co-ordinates where always going to be approximation anyway.
What was misleading is that my current approach started off so well, and the actual predictions I was getting on test data where wrong, but they were not unreasonable, you could see the logic.
Any thoughts would be appreciated?
Dave