Maybe you could just calculate the indices and slice the input tensor directly:
N, C, H, W = 1, 3, 10, 10
h, w = 5, 5
x = torch.randint(0, 10, (N, C, H, W))
iw = torch.linspace(0, W-1, w).long()
ih = torch.linspace(0, H-1, h).long()

x_interp = x[:, :, ih[:, None], iw]

Would that work for you?