@TG_N Yes! That is a great idea!
In fact, this is how a number of convolutional neutral nets for image registration and optical flow are structured.
The CNN predicts a flow/grid/mapping/displacement field which is then passed to grid_sample() or the like (you may have to do some conversions to fit the conventions), and then backpropogate from the result.