Solved by kairos03 in post #5 
              
              
                One of my friend solve this problem. 
The solution is expanding index matrix and use gather 
# for 3-dim input
X = torch.tensor([[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 10], [11, 12]]])
I = torch.tensor([[0, 2], [2, 1]])
eI = I[..., None].expand(-1, -1, X.size(2))    ## expanding index
Y = torch.gath…