This seems to work, but then after implementing it and trying to train my model with it I seem to get this error:
  File "train.py", line 252, in upsnet_train
    loss.backward()
  File "envs/pytorch1.1/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "envs/pytorch1.1/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: invalid configuration argument

Note, after I interpolate the points, I compute a loss on top of it. The interpolation code is here:
B, N_old, N_new = points.size(0), points.size(1), points.size(1) * 2
grid = torch.cat([torch.zeros((B, N_new, 1, 1)),
                  torch.linspace(0, N_old, N_new).unsqueeze(0).repeat((B, 1)).unsqueeze(
                      -1).unsqueeze(-1)], dim=-1).cuda()
grid = (grid - N_old/2.) / N_old * 2.
rows = F.grid_sample(points.unsqueeze(1)[:, :, :, 0].unsqueeze(-1), grid).squeeze(1)
cols = F.grid_sample(points.unsqueeze(1)[:, :, :, 1].unsqueeze(-1), grid).squeeze(1)
points = torch.cat([rows,cols], dim=-1)