This code might work:
idx = torch.tensor([[[[0, 1],
                      [1, 2],
                      [2, 3],
                      [3, 0]],
                     [[0, 1],
                      [1, 2],
                      [2, 3],
                      [3, 0]],
                     [[1, 0],
                      [0, 2],
                      [2, 3],
                      [3, 1]]],
                    [[[0, 0],
                      [1, 1],
                      [2, 2],
                      [3, 3]],
                     [[0, 0],
                      [1, 1],
                      [2, 2],
                      [3, 3]],
                     [[3, 3],
                      [2, 2],
                      [1, 1],
                      [0, 0]]]])

x = torch.randn(2, 4, 4)
x = x.unsqueeze(1).expand(-1, idx.size(1), -1, -1)
res = x[torch.arange(x.size(0))[:, None, None, None],
        torch.arange(x.size(1))[:, None, None],
        idx[:, :, :, 0:1], idx[:, :, :, 1:2]]

Could you verify it, as I was too lazy to type in your distance matrices?