Solved by koelscha in post #2 
              
              
                I guess, I could do it by expanding the indices and calling gather twice, as follows: 
x = b[:,:,0]
x = x.view(4,16,1,1).expand(4,16,256,1)
y = b[:,:,1]
y = y.view(4,16,1,1)
result = a.gather(3, x).gather(2, y).squeeze()

but I think this looks a bit clumsy and I feel like there should be a better w…