The shift vector is computed in the paper as given below

Screenshot from 2020-04-28 17-52-24830×462 52.8 KB

the idea is basically to perform the nearest neighbour search for each pixel in the corrupted region and then replace the nearest neighbour element inplace and further concatenating with the already upsampled and skip connected feature map.
But I am unable to figure out a way to do so in pytorch!