Something like the following works:
a = torch.FloatTensor([[0.1, 0.3, 0.5, 0.1],
                       [0.2, 0.2, 0.3, 0.3]])

a[ [[0, 0], [1, 1]] , [[0, 1], [2, 3]] ]

so it looks like you’d want to coax b into something that looks like [[0, 0], [1, 1]] , [[0, 1], [2, 3]]