Thanks, I see it. torch.gather is unuseful here I feel.I use torch_select.
t = torch.Tensor([[1,2,3],[3,4,5], [7,9,9]])
print(t)
torch.cat((torch.index_select(t, 1, torch.LongTensor([0,2])), torch.index_select(t, 1, torch.LongTensor([1]))), dim=1)


 1  2  3
 3  4  5
 7  9  9
[torch.FloatTensor of size 3x3]

Out[66]:

 1  3  2
 3  5  4
 7  9  9
[torch.FloatTensor of size 3x3]