You could try to add additional channels containing the coordinate information as described in this paper from Uber 5.
Here is also the corresponding video explanation:
An Intriguing Failing of Convolutional Neural Networks and the CoordConv...