The solution is simple:
reshape the grid as (1 X NOH x OW x2)
call grid_sample and reshape the output to (NxCxOHxOw)!
The downside is that you may have border issues due to the interpolation of coordinates in very different places…