Hey guys,
feel free to correct me, but I think this might be the correct solution:
batch_size = 16
c, h, w = 256, 16, 16
nb_points = 150
img_feat = torch.randn(batch_size, c, h, w)
x = torch.empty(batch_size, nb_points, dtype=torch.long).random_(h)
y = torch.empty(batch_size, nb_points, dtype=torch.long).random_(w)
x = x[:, None, :, None].expand(-1, c, -1, w)
y = y[:, None, :, None].expand(-1, c, -1, 1)

points = torch.gather(torch.gather(img_feat, 2, x), 3, y)

This way, it should collect only 1 (hopefully the correct ;)) value for each of the nb_points * w extracted values from the first gather in the second gather step.
That should be correct, right?