Solved by SimonW in post #2 
              
              
                N = 10
C = 3
H = 5
W = 5
x = torch.randn(N, C, H, W)
indices = torch.randint(H * W, (N, C, 1), dtype=torch.long)
torch.gather(x.view(N, C, H * W), 2, indices).view(N, C)