I’m not sure how your indexing works, since I wasn’t able to run your code.
Based on the shapes, would something like this work?
B, N, C, M = 2, 3, 4, 5
x = torch.randn(B, N, C)
idx = torch.randint(0, C, (B, N, M))
torch.gather(x, 2, idx)